Click here to go to the forum index Click here for the home page
 
Author Message

<  New users start here  ~  Recording External Sources

Page 2 of 2
Goto page Previous  1, 2
EMJB
Posted: Thu Oct 27, 2011 7:06 pm Reply with quote
Frequent contributor Joined: 08 Jul 2005 Posts: 3645 Location: Maldon Essex
Monode wrote:
Precisely, thats why it does defragging on the fly.

The very nature of a file structure means any drive will get fragmented. Whether you are dealing in 1k files or multi-exabyte slabs, its all the same thing.


I think you are overlooking the fact that a PVR drive is writing a few files a day, and (TAP activities excluded) never extending files one they are written. Also files are mostly very large (recordings) or less than the size of a sector (ini files). Fragmentation will there occur at a very much lower rate than a hard drive used for storing correspondence etc.

EMJB
View user's profile Send private message
Monode
Posted: Thu Oct 27, 2011 8:18 pm Reply with quote
Joined: 18 Oct 2011 Posts: 11
Okay, I do 3D animation. Some years ago, probably 10 now, we sold Video Toaster which required fast, noisy, expensive Ultra SCSI drives in a RAID 5 to make it work. Each channel of uncompressed TV requires 26Mb of data per second sustained.

We knew one guy who ran it off two ATA66 drives striped together under NT and it worked fine. This is software RAIDing.

Modern drives can pump out serious amounts of data so with compressed MPEG2 data pretty much any single modern drive will perform. SSD is in another league again - those things flood the connection at 6Gbits per second!

Drives have come a long way so even non-'AV' drives (which don't have to thermally recalibrate) can playback a channel while recording two others and find time to run system updates, thermal calibration and FAT ops.

Sorry - 30 years in computing, you have a lots useless data in the back of your brain! :o)
View user's profile Send private message
gomezz
Posted: Thu Oct 27, 2011 11:14 pm Reply with quote
Frequent contributor Joined: 18 Nov 2005 Posts: 3996 Location: Buckingham
But what has all that to do with the easy time a PVR gives a hard disk?

(26 years in banking IT allocating ledgers across multiple disk arrays making sure the index files are allocated to the limited amount of SSD available to maximise transaction thoughput by spreading the workload across all available I/O channels and disks)

_________________
TF5800, F/W: MS6 Recommended F/W 12/9/2009 -Sy+BmC0CbCfCtDEgEmEvEzIMPePsVdZ
TAPs: Power Down 0.75; QuickJump 1.72; MyStuff 6.6; TAP Commander 1.34; MyInfo B5.4; EPG2MEI v0.96;
Extend v1.7; EIT Sub v0.6; mei2archive BETA 3.8l6; Font Manager 1.0d; (Power Restore V0.7.3b)
View user's profile Send private message Visit poster's website
Monode
Posted: Fri Oct 28, 2011 1:58 am Reply with quote
Joined: 18 Oct 2011 Posts: 11
Gomezz - With all respect, what you are saying is fragmentation still happens. I know people in their 20s who record a dozen shows plus a week and lets not forget that the TiVo looks at what you likes, runs the titles through a database and finds where similar programmes are on and records them automagically so your regular Gadget Show, Top Gear and some cookery show soon becomes anything in those genres. Pretty quickly the hard drive gets full and the less important stuff gets wiped for higher starred content.

This really was an issue and, much to my mates annoyance, they let an empoverished 17 year old Belgium art student stay for free with them and discovered she had starred every bit of schmaltzy crap on TV so they had to format the hard drive and reset the BIOS every week! :D Good thing he was a developer with God like privileges - TiVo grows with you and back then resetting the 'likes' was tedious.

I see your point, however TiVo's have a different operational brief and thus the need for on-the-fly defrag. Its why TiVo won over rivals, well, that and aggressive marketing - see the film Tropic Thunder.
View user's profile Send private message
DX
Posted: Fri Oct 28, 2011 10:22 am Reply with quote
Frequent contributor Joined: 06 Apr 2005 Posts: 2695
Whie it is true that today PVRs write data in large contiguous chunks that don't need periodic defragmentation people should consider how and why they work that way, it isn't an accident.

Back in the 1990's when TiVo was being developed the linux file systems in use at the time did suffer badly from fragmentation. They also had a large CPU overhead in doing the I/O.

Besides the fragmentation issue there was also a problem of robustness in a domestic environment - traditional linux files systems needed to be shutdown cleanly and didn't appreciate users powering off at random times. A PVR needs a file system that can survive such treatment.

So TiVo developed a Media File System (MFS) that was robust and didn't suffer from fragmentation.
View user's profile Send private message
Monode
Posted: Fri Oct 28, 2011 1:29 pm Reply with quote
Joined: 18 Oct 2011 Posts: 11
God yeah, I'd forgotten how like running a traction engine Linux was back then actually. Nice post! :o)
View user's profile Send private message
R2-D2
Posted: Fri Oct 28, 2011 7:55 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
DX wrote:
So TiVo developed a Media File System (MFS) that was robust and didn't suffer from fragmentation.

No fragmentation? Or just the same completely insignificant fragmentation that using a very large cluster size (like the Toppy's filesystem) implies? [By "very large" we're talking around 1000 times bigger than that used in a normal PC's disk which also has to cope with loads of small files.]

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
Monode
Posted: Fri Oct 28, 2011 10:10 pm Reply with quote
Joined: 18 Oct 2011 Posts: 11
I feel those regurgitating 'Big files don't get fragmentation' need to reconsider their tenure of the subject. The very fact that DX wrote such a poignantly clear and concise description and I happen to know from personal experience that a new company was amply prepared to invest considerable sums of development costs into resolving technical issues still does not seem to abate the unmitigated diatribe.
View user's profile Send private message
DX
Posted: Fri Oct 28, 2011 10:45 pm Reply with quote
Frequent contributor Joined: 06 Apr 2005 Posts: 2695
R2-D2 wrote:
DX wrote:
So TiVo developed a Media File System (MFS) that was robust and didn't suffer from fragmentation.

No fragmentation? Or just the same completely insignificant fragmentation that using a very large cluster size (like the Toppy's filesystem) implies? [By "very large" we're talking around 1000 times bigger than that used in a normal PC's disk which also has to cope with loads of small files.]

See

http://en.wikipedia.org/wiki/Tivo's_Media_File_System
View user's profile Send private message
gomezz
Posted: Fri Oct 28, 2011 11:11 pm Reply with quote
Frequent contributor Joined: 18 Nov 2005 Posts: 3996 Location: Buckingham
Monode wrote:
I feel those regurgitating 'Big files don't get fragmentation' need to reconsider their tenure of the subject.

That shows you are not listening. We are saying PVRs do not suffer from defragmentation in the same way that computers do which is different to saying they do not undergo defragmentation. If Tivo have had to work hard to adapt an unsuitable filing system to make it work for them then it is curious that no other PVR maker has needed to to do the same.

_________________
TF5800, F/W: MS6 Recommended F/W 12/9/2009 -Sy+BmC0CbCfCtDEgEmEvEzIMPePsVdZ
TAPs: Power Down 0.75; QuickJump 1.72; MyStuff 6.6; TAP Commander 1.34; MyInfo B5.4; EPG2MEI v0.96;
Extend v1.7; EIT Sub v0.6; mei2archive BETA 3.8l6; Font Manager 1.0d; (Power Restore V0.7.3b)
View user's profile Send private message Visit poster's website
simonc
Posted: Fri Oct 28, 2011 11:43 pm Reply with quote
Frequent contributor Joined: 12 Apr 2005 Posts: 5642 Location: Cheltenham
Interesting link. Turns out the Tivo has a video cluster size the same as a 160GB Toppy.

If your OS is based on a consumer OS that loads from disk then there might be a need to defragment the OS partition to maintain a consistent performance, e.g. boot speed. Were drives sufficiently borderline 10 years ago to impact video read rates? Perhaps it was a commercial decision to invest in the one off job of writing the defragmenter rather than buy lots of more expensive disks with larger caches to avoid video stutter whilst the OS partition was being read.
View user's profile Send private message Visit poster's website
Monode
Posted: Sat Oct 29, 2011 1:35 am Reply with quote
Joined: 18 Oct 2011 Posts: 11
My Rainbow Runner was playing back near broadcast quality video from IDE and SCSI-2 HDDs from my Pentium 150 machine back in 1997!

TiVo is using compressed video whereas the Video Toaster is a professional system that can deal with multiple streams of uncompressed HD thesedays. That said, two ATA66 (I have a feeling it was ATA33 though) could deliver 26Mb a second sustained as a striped pair for real time PAL. SATA is so much quicker that I wonder where that is going to lead - you can imagine HDTVs with SSD recorders built in before long and oh how the broadcasters will love a sealed-system like that.
View user's profile Send private message
R2-D2
Posted: Sat Oct 29, 2011 10:46 am Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
DX wrote:
See

http://en.wikipedia.org/wiki/Tivo's_Media_File_System


So the same as the Toppy and still scope for plenty of fragmentation but all of it completely insignificant given the cluster size (and a disk with a sensible number of heads, cache and seek rate).

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
DX
Posted: Sun Oct 30, 2011 11:46 am Reply with quote
Frequent contributor Joined: 06 Apr 2005 Posts: 2695
I do wonder where this thread is going Confused

I assume many here (if not most) are familiar with TopfHDRW? It exists because Topfield developed a non-standard file system for pvr use. That non-standard file system is FAT based and uses large cluster sizes which mitigate the effects of file fragmentation.

If Topfield felt the need to develop their own file system why do some people have such a hard time accepting that other manufacturers might have felt a similar need but arrived at a different solution?

There has been considerable effort put into file system development over the past 15 years by different groups.

The Linux ext2 files system has morphed into ext3 and then ext4. These address many of the issues a pvr manufacturer would have faced in the mid 1990's.

Silcon Graphics developed XFS in the 1990's. As you can see from this link

http://en.wikipedia.org/wiki/XFS

"XFS is particularly proficient at handling large files and at offering smooth data transfers."

It was released under GPL in 2000 and is now used in a number of pvrs.

As a result of the efforts of many people over many years you can today get an "off-the-shelf" file system suitable for use in a pvr. This doesn't mean however that Topfield and TiVo were idiots for developing their own file systems. At the time they had to. Nor does it mean there is a right way or a wrong way, both came up solutions that worked for them.

I do hope we can move on from this idea there is only one way to build a pvr.
View user's profile Send private message

Display posts from previous:  

All times are GMT + 1 Hour
Page 2 of 2
Goto page Previous  1, 2

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum