For questions, answers and tips aimed at new users, or those thinking about buying a Toppy

Moderators: General, Technical

EMJB
Frequent contributor
Posts: 3645
Joined: Fri Jul 08, 2005 9:43 am
Location: Maldon Essex

Post by EMJB »

Monode wrote:Precisely, thats why it does defragging on the fly.

The very nature of a file structure means any drive will get fragmented. Whether you are dealing in 1k files or multi-exabyte slabs, its all the same thing.
I think you are overlooking the fact that a PVR drive is writing a few files a day, and (TAP activities excluded) never extending files one they are written. Also files are mostly very large (recordings) or less than the size of a sector (ini files). Fragmentation will there occur at a very much lower rate than a hard drive used for storing correspondence etc.

EMJB
Monode
Posts: 11
Joined: Tue Oct 18, 2011 1:31 am

Post by Monode »

Okay, I do 3D animation. Some years ago, probably 10 now, we sold Video Toaster which required fast, noisy, expensive Ultra SCSI drives in a RAID 5 to make it work. Each channel of uncompressed TV requires 26Mb of data per second sustained.

We knew one guy who ran it off two ATA66 drives striped together under NT and it worked fine. This is software RAIDing.

Modern drives can pump out serious amounts of data so with compressed MPEG2 data pretty much any single modern drive will perform. SSD is in another league again - those things flood the connection at 6Gbits per second!

Drives have come a long way so even non-'AV' drives (which don't have to thermally recalibrate) can playback a channel while recording two others and find time to run system updates, thermal calibration and FAT ops.

Sorry - 30 years in computing, you have a lots useless data in the back of your brain! :o)
gomezz
Frequent contributor
Posts: 4000
Joined: Fri Nov 18, 2005 6:45 pm
Location: Buckingham
Contact:

Post by gomezz »

But what has all that to do with the easy time a PVR gives a hard disk?

(26 years in banking IT allocating ledgers across multiple disk arrays making sure the index files are allocated to the limited amount of SSD available to maximise transaction thoughput by spreading the workload across all available I/O channels and disks)
TF5800, F/W: MS6 Recommended F/W 12/9/2009 -Sy+BmC0CbCfCtDEgEmEvEzIMPePsVdZ
TAPs: Power Down 0.75; QuickJump 1.72; MyStuff 6.6; TAP Commander 1.34; MyInfo B5.4; EPG2MEI v0.96;
Extend v1.7; EIT Sub v0.6; mei2archive BETA 3.8l6; Font Manager 1.0d; (Power Restore V0.7.3b)
Monode
Posts: 11
Joined: Tue Oct 18, 2011 1:31 am

Post by Monode »

Gomezz - With all respect, what you are saying is fragmentation still happens. I know people in their 20s who record a dozen shows plus a week and lets not forget that the TiVo looks at what you likes, runs the titles through a database and finds where similar programmes are on and records them automagically so your regular Gadget Show, Top Gear and some cookery show soon becomes anything in those genres. Pretty quickly the hard drive gets full and the less important stuff gets wiped for higher starred content.

This really was an issue and, much to my mates annoyance, they let an empoverished 17 year old Belgium art student stay for free with them and discovered she had starred every bit of schmaltzy crap on TV so they had to format the hard drive and reset the BIOS every week! :D Good thing he was a developer with God like privileges - TiVo grows with you and back then resetting the 'likes' was tedious.

I see your point, however TiVo's have a different operational brief and thus the need for on-the-fly defrag. Its why TiVo won over rivals, well, that and aggressive marketing - see the film Tropic Thunder.
DX
Frequent contributor
Posts: 2697
Joined: Wed Apr 06, 2005 4:09 pm

Post by DX »

Whie it is true that today PVRs write data in large contiguous chunks that don't need periodic defragmentation people should consider how and why they work that way, it isn't an accident.

Back in the 1990's when TiVo was being developed the linux file systems in use at the time did suffer badly from fragmentation. They also had a large CPU overhead in doing the I/O.

Besides the fragmentation issue there was also a problem of robustness in a domestic environment - traditional linux files systems needed to be shutdown cleanly and didn't appreciate users powering off at random times. A PVR needs a file system that can survive such treatment.

So TiVo developed a Media File System (MFS) that was robust and didn't suffer from fragmentation.
Monode
Posts: 11
Joined: Tue Oct 18, 2011 1:31 am

Post by Monode »

God yeah, I'd forgotten how like running a traction engine Linux was back then actually. Nice post! :o)
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

DX wrote:So TiVo developed a Media File System (MFS) that was robust and didn't suffer from fragmentation.
No fragmentation? Or just the same completely insignificant fragmentation that using a very large cluster size (like the Toppy's filesystem) implies? [By "very large" we're talking around 1000 times bigger than that used in a normal PC's disk which also has to cope with loads of small files.]
Monode
Posts: 11
Joined: Tue Oct 18, 2011 1:31 am

Post by Monode »

I feel those regurgitating 'Big files don't get fragmentation' need to reconsider their tenure of the subject. The very fact that DX wrote such a poignantly clear and concise description and I happen to know from personal experience that a new company was amply prepared to invest considerable sums of development costs into resolving technical issues still does not seem to abate the unmitigated diatribe.
DX
Frequent contributor
Posts: 2697
Joined: Wed Apr 06, 2005 4:09 pm

Post by DX »

R2-D2 wrote:
DX wrote:So TiVo developed a Media File System (MFS) that was robust and didn't suffer from fragmentation.
No fragmentation? Or just the same completely insignificant fragmentation that using a very large cluster size (like the Toppy's filesystem) implies? [By "very large" we're talking around 1000 times bigger than that used in a normal PC's disk which also has to cope with loads of small files.]
See

http://en.wikipedia.org/wiki/Tivo's_Media_File_System
gomezz
Frequent contributor
Posts: 4000
Joined: Fri Nov 18, 2005 6:45 pm
Location: Buckingham
Contact:

Post by gomezz »

Monode wrote:I feel those regurgitating 'Big files don't get fragmentation' need to reconsider their tenure of the subject.
That shows you are not listening. We are saying PVRs do not suffer from defragmentation in the same way that computers do which is different to saying they do not undergo defragmentation. If Tivo have had to work hard to adapt an unsuitable filing system to make it work for them then it is curious that no other PVR maker has needed to to do the same.
TF5800, F/W: MS6 Recommended F/W 12/9/2009 -Sy+BmC0CbCfCtDEgEmEvEzIMPePsVdZ
TAPs: Power Down 0.75; QuickJump 1.72; MyStuff 6.6; TAP Commander 1.34; MyInfo B5.4; EPG2MEI v0.96;
Extend v1.7; EIT Sub v0.6; mei2archive BETA 3.8l6; Font Manager 1.0d; (Power Restore V0.7.3b)
simonc
Frequent contributor
Posts: 5648
Joined: Tue Apr 12, 2005 1:31 pm
Location: Cheltenham
Contact:

Post by simonc »

Interesting link. Turns out the Tivo has a video cluster size the same as a 160GB Toppy.

If your OS is based on a consumer OS that loads from disk then there might be a need to defragment the OS partition to maintain a consistent performance, e.g. boot speed. Were drives sufficiently borderline 10 years ago to impact video read rates? Perhaps it was a commercial decision to invest in the one off job of writing the defragmenter rather than buy lots of more expensive disks with larger caches to avoid video stutter whilst the OS partition was being read.
Monode
Posts: 11
Joined: Tue Oct 18, 2011 1:31 am

Post by Monode »

My Rainbow Runner was playing back near broadcast quality video from IDE and SCSI-2 HDDs from my Pentium 150 machine back in 1997!

TiVo is using compressed video whereas the Video Toaster is a professional system that can deal with multiple streams of uncompressed HD thesedays. That said, two ATA66 (I have a feeling it was ATA33 though) could deliver 26Mb a second sustained as a striped pair for real time PAL. SATA is so much quicker that I wonder where that is going to lead - you can imagine HDTVs with SSD recorders built in before long and oh how the broadcasters will love a sealed-system like that.
R2-D2
Frequent contributor
Posts: 12148
Joined: Mon Dec 18, 2006 11:15 am
Contact:

Post by R2-D2 »

So the same as the Toppy and still scope for plenty of fragmentation but all of it completely insignificant given the cluster size (and a disk with a sensible number of heads, cache and seek rate).
DX
Frequent contributor
Posts: 2697
Joined: Wed Apr 06, 2005 4:09 pm

Post by DX »

I do wonder where this thread is going :?

I assume many here (if not most) are familiar with TopfHDRW? It exists because Topfield developed a non-standard file system for pvr use. That non-standard file system is FAT based and uses large cluster sizes which mitigate the effects of file fragmentation.

If Topfield felt the need to develop their own file system why do some people have such a hard time accepting that other manufacturers might have felt a similar need but arrived at a different solution?

There has been considerable effort put into file system development over the past 15 years by different groups.

The Linux ext2 files system has morphed into ext3 and then ext4. These address many of the issues a pvr manufacturer would have faced in the mid 1990's.

Silcon Graphics developed XFS in the 1990's. As you can see from this link

http://en.wikipedia.org/wiki/XFS

"XFS is particularly proficient at handling large files and at offering smooth data transfers."

It was released under GPL in 2000 and is now used in a number of pvrs.

As a result of the efforts of many people over many years you can today get an "off-the-shelf" file system suitable for use in a pvr. This doesn't mean however that Topfield and TiVo were idiots for developing their own file systems. At the time they had to. Nor does it mean there is a right way or a wrong way, both came up solutions that worked for them.

I do hope we can move on from this idea there is only one way to build a pvr.
Post Reply