Click here to go to the forum index Click here for the home page
 
Author Message

<  MyStuff  ~  Incorrect Recovery Mode/Poor additions to RecordedInfo

Page 1 of 1
Geoff Bacon
Posted: Mon Jul 11, 2016 8:04 pm Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
I doubt that we will get another update to MyStuff but just in case.

Problem 1
MS enters recovery mode when the toppy boots and reads an EPG.mei file that doesn't contain any current events e.g. when the toppy has been switched on for a couple of weeks.
This catches me every time because, from the error message, it appears that Mystuff has crashed whereas all it needs to do is wait for EPG2MEI to generate a file (as it would if the file didn't exist at all).

Problem 2
I'm trying to get my disk recovery program (TDM) to automatically add entries into the MS_RecordedInfo.dat file when a recording is added to the disk (currently, when many recordings need to be added, Mystuff takes "an eternity" and the loading icon looks like MS is stuck. Experimenting with a single file, I see that the entry added to the file is given the date of the file in the file system (fair enough). However, it also modifies the start and end times relative to this file date - it should really maintain the values embedded in the recording header.
ISTR that some members (but can't find the posts) had to downgrade to MS 6.4 because, after some sort of external recovery process, all their recordings were recovered with an identical date. My guess is that 6.4 populated the file with start and end times taken from the header.

Geoff

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website
mstombs
Posted: Tue Jul 12, 2016 8:41 am Reply with quote
Frequent contributor Joined: 31 Dec 2006 Posts: 938
Re issues, I recently had a repeat of a problem with a power search for "Big Bang Theory", it kept picking up repeats which I manually excluded, I assume this is broadcaster issue re inconsistent use of series/episode crid info. Finally the power search turned into a text search 'flagging' all future episodes (48 in next few days!). Couldn't edit the search, so have just deleted. Suspect there is a limit somewhere on maximum number of episodes or excludes?
View user's profile Send private message TF5800
YorkshireJumbo
Posted: Tue Jul 12, 2016 8:57 am Reply with quote
Frequent contributor Joined: 16 Sep 2005 Posts: 837
@mstombs I've had the same with Rude Tube, and also when we used to record You've Been Framed a few years ago. ISTR it's to do with the number of different series that it is trying to keep track of. When there are too many, it flips it to a Flag Search. I've always been able to change it back to a PS, though YBF failed again a few weeks later. I fixed that by tweaking the FS so it wouldn't find anything, creating a Series Search and reinstating the Flag Search to find new episodes.

_________________
YJ - You may have speed, but I have momentum
Black Panther since Sep '05, Caps replaced July '10
WD10EZRX and Turbosat adaptor added May '16
Noiseblocker XR1 fan added Mar '18
F/W: MS6 Recommended F/W 12/9/2009 -Sy+U+Uu
TAPs: MS 6.6 Suite, UK OZ Surfer
View user's profile Send private message
andyfras
Posted: Tue Jul 12, 2016 9:06 am Reply with quote
Frequent contributor Joined: 25 Jul 2005 Posts: 3477
Geoff Bacon wrote:
ISTR that some members (but can't find the posts) had to downgrade to MS 6.4 because, after some sort of external recovery process, all their recordings were recovered with an identical date.

That was me:

http://forum.toppy.org.uk/forum/viewtopic.php?p=259987#259987

_________________
Toppy PSU repair service or capacitor kits for DIY available - PM me for details.
MyStuff 6.6 : Little Clock : MHEGOnOff : Extend : EPG2MEI : SecCache : EIT_Sub : TSSaver : Prune EPG
Channel Organiser : Tap Commander : HDD Info : PCControl : Signal Monitor : MyInfo
URC-7555 remote: TF5800 320GB 2.5", Recommended_5800.tfd + CfPePsScUUuWfZ
View user's profile Send private message Visit poster's website
Geoff Bacon
Posted: Tue Jul 12, 2016 10:40 am Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
I knew someone had reported it (and should have guessed that it was you!)

The problem occurs because MS is using just the Channel Name and the file creation date to find the matching record in the recorded info file.

A better approach (MEI9?) would be to checksum the first few bytes of the header and use that to identify the resulting record. One would have to watch out for things that might change in the header e.g. the Lock bit and the bookmarks - perhaps some others (these could be forced to a fixed value prior to computing the checksum). As the header may contain variant records indicating the source of the recording, one could perhaps just checksum the bytes occupied by the smallest variant of each type (and excluding the bookmarks at the end)
The first two fields in the RecordedInfo file would be "checksum.mei|MEI9|"

This would probably need to be combined with the cluster number of the file on the disk (I can provide sample code to identify the cluster) so that on a reindex, MS can easily identify files that have appeared. This wouldn't affect the MEI format as MS could do this solely in memory i.e. the cluster number wouldn't need to be stored in the RecordedInfo file.

In the Archive, MS would have to display the start time (from the header) rather than the file creation date (as at present)

Using a checksum would still work if a file was transferred to the toppy from a pc. It wouldn't match the record if someone had changed the header using another program (e.g. by changing the stored description) but then MS should just detect it as though it was an unknown file and build a new entry for it.

Note: FTP doesn't normally maintain the source date of a file that it transfers. This means that any files transferred by FTP between a toppy and pc will end up with the same time stamp (at least to the minute). This will cause the problem unless there is at most one recording per channel.
i.e. Don't use FTP to copy the files unless it maintains the source date on the output file.


I can understand why BobD might not what to start another round of programming!

As far as TDM is concerned, I'm looking at modifying the program to force the creation dates of recordings to match that stored in their recording headers - but this won't help anyone else trying to do it themselves. Although I can do this, trying to match/remove an "incorrect" entry in the RecordingInfo file is not proving easy.

Geoff

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website
mstombs
Posted: Tue Jul 12, 2016 2:28 pm Reply with quote
Frequent contributor Joined: 31 Dec 2006 Posts: 938
@GB Please do not propose anything that slows down the MyStuff Loading phase (takes minutes on 1.5TB disk with >1000 recordings!) - but I think it must read the header of every file to get the Channel No at the moment, to match info in RecordedInfo so possible to implement some sort of caching/ quick decision on change?
View user's profile Send private message TF5800
Geoff Bacon
Posted: Tue Jul 12, 2016 3:29 pm Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
I'm not proposing anything that would slow down the loading phase - I'm just suggesting how it could be done by using the current mechanism with a change of the data stored.

Caching the data is a difficult thing; it would be easy if the user couldn't manipulate files outside MS!

If I was implementing caching, I would try to proceed as follows:
1) MS would need to save the current FAT array when it closes (store as binary, very quick)
2) The MEI format would be modified to have "ClusterNumber_FatChecksum" as the first entry (instead of ChannelName_FileCreationDate)
3) When loading, MS would read the RecordedInfo file and the current "Live" FAT array table i.e. take a snapshot of the FAT.
4) MS would scan the directories looking for recordings by using code that returned the cluster number for the start of the recording (this number is unique)
5) The cluster number would be used to locate the entry corresponding to this recording.
6) MS would follow the chain of clusters (in the snapshot of the "Live" FAT) and store in a vector. It would checksum this vector and, if it matched that in the recordedinfo record, it would assume that the stored info was valid. If it didn't match, it would create a new entry (much as it would as present).
Note: a checksum match would mean that MS didn't need to read the recording header during the loading phase.
7) When playing a recording entry, MS would sanity check (and amend) the ClusterNumber_CheckSum pair (just in case someone has truncated the file). This would allow the entry details to still be maintained even though the file has been truncated.


Comments on the above
a) Copying files from a pc to the toppy will always result in a new entry (before, it may have picked up an existing entry because it matched the channel and file stamp). This means that entries for copied files would never contain data that cannot be gleaned from the recording header e.g. crids
b) Moving files between folders should not require manipulation of the entry (but this would need testing). This is because the file would still reside in the same clusters, it would just be listed in a a different folder.

There are probably a few other things one would have to worry about but I can only guess at these because I don't know how MS is coded.

I can't see the above ever being implemented unless BobD gets bored and fancies a challenge.

Geoff

[edit]Rereading the above, I see that there is no need to save the live FAT array when MS shuts down; my original idea was that one could compare the cluster chains in the two arrays but this is obviated bu using checksums [/edit]

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website
mstombs
Posted: Tue Jul 12, 2016 8:43 pm Reply with quote
Frequent contributor Joined: 31 Dec 2006 Posts: 938
Offline in php MWI uses the folder information name, size, date to create a hash

Quote:
$hash = sha1($fullpath.$recdate.$size);
View user's profile Send private message TF5800
Geoff Bacon
Posted: Tue Jul 12, 2016 10:08 pm Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
I think BobD is trying to avoid using the name in the key - the user can easily change it or move it into another folder, The existing mechanism would handle these cases.

Geoff

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website
juwlz
Posted: Wed Jul 13, 2016 10:56 am Reply with quote
MyStuff Team Joined: 12 Aug 2005 Posts: 10804 Location: Wokingham, Berkshire (Hannington transmitter)
Geoff Bacon wrote:
I doubt that we will get another update to MyStuff but just in case.
You're right. Here's BobD's response to a recent minor request: http://forum.toppy.org.uk/forum/viewtopic.php?p=269731#269731

Julie

_________________
5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+BmC0CfCtFsIMPePsR3UUuUxZ
5810, TS On, F/W: F/W: MS6 Recommended F/W 12/9/2009 +BmCfCtFsR3Z
MyStuff info; Help!; Reference
Harmony 885 remote. Sig date 8 April 2012
View user's profile Send private message Visit poster's website
BobD
Posted: Wed Jul 13, 2016 2:32 pm Reply with quote
MyStuff Team Joined: 03 Aug 2005 Posts: 4218
Yup, I'm unlikely to get enough enthusiasm up for any change for a while, unless it is a spectacularly good new feature. Something like this (which may make sense, but confused me very early on in the numbered list!) is not very likely to see the light of day Smile. Sorry.

Modifying the RecordedInfo.dat should work though. Is it not possible to change the file creation date when doing restores, and then rewrite the RecordedInfo to match?

_________________
FW: ChunkyWizard Recommended
TAPs:
MyStuff (always one version ahead of everyone else!), and recommended support TAPS
MyStuff skins, manual and latest version: http://www.BobDsMyStuff.co.uk
Known bugs & forthcoming fixes: http://www.BobDsMyStuff.co.uk/Bugs.shtml
Changes coming in the next version: http://www.BobDsMyStuff.co.uk/NextVersion.shtml
View user's profile Send private message Visit poster's website
Geoff Bacon
Posted: Wed Jul 13, 2016 3:39 pm Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
@BobD

Thanks for the response.
Quote:
Modifying the RecordedInfo.dat should work though. Is it not possible to change the file creation date when doing restores, and then rewrite the RecordedInfo to match?
That is what I am attempting to do but I'm having difficulty matching any existing records in the Recordedinfo file - it doesn't contain recording dates that match those in recording header.

Example For a recording of one hour at 21:00 on 13/04/2015 the header record contains Start=201504132100 and end= 201504132200, (Note: I have invented these numbers as I no longer have the relevant output file).

Copying the file over results in an RecordedInfo entry of
BBC TWO_201607121600.mei|0|MEI8|201607121600|201607121700|...
i.e. it has changed the start and end times to be relative to the first field. I would have no problems identifying the record if it had maintained the original recording info and had been written as
BBC TWO_201607121600.mei|0|MEI8|201504132100|201504132200|...

The issue only arises for files that RecordedInfo already knows (changing dates for "discovered" files is not a problem because they are not cataloged in RecordedInfo).

I can probably devise a method to match based on the recording description (slightly complicated by perhaps having to allow for the Event Name and Event Number having been extracted into separate fields in the RecordedInfo entry). Any such algorithm also assumes that the Description is unique (not much use if a programme is broadcast regularly and uses a common description across all the recordings).

Anyway, to summarize, I'm not expecting a new version of MS; I'm just documenting that the MEI8 mechanism has it's drawbacks. The thread then got convoluted by how MS could cache data so that it would load quicker.

Like you, I don't think there is much point in spending a great deal of time improving our toppy program/utilities - I'm really just doing it for my own amusement.

Many thanks for all the work and effort you have spect writing MyStuff.

Cheers
Geoff

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website
BobD
Posted: Wed Jul 13, 2016 4:09 pm Reply with quote
MyStuff Team Joined: 03 Aug 2005 Posts: 4218
Geoff Bacon wrote:
Example For a recording of one hour at 21:00 on 13/04/2015 the header record contains Start=201504132100 and end= 201504132200, (Note: I have invented these numbers as I no longer have the relevant output file).

Copying the file over results in an RecordedInfo entry of
BBC TWO_201607121600.mei|0|MEI8|201607121600|201607121700|...
i.e. it has changed the start and end times to be relative to the first field. I would have no problems identifying the record if it had maintained the original recording info and had been written as
BBC TWO_201607121600.mei|0|MEI8|201504132100|201504132200|...


OK, let me see if I follow this.

I am assuming that the problem is caused by the file creation date on the .rec being reset to the current date when you copy it back to the toppy?

So MyStuff finds the old recording, but with the new date. It then discovers that there is no entry in the RecordedInfo file that matches this recording (because the Info file has the original/correct date), and so then creates a new entry for it (now with the wrong date).

Is that correct?

I do remember changing something at some point (in order to fix a bug where recordings that happen when MyStuff was not running did not get listed correctly) which would have caused this to happen. I think. I could probably change that to use the date & time in the recording header instead (although I try to avoid using too much from the header so as to make my life easier when coping with different toppy file formats).

As for the recovery mode problem you mention, the lack of EPG data in the file shouldn't trigger recovery mode. Ah, maybe it does. I was thinking of the other message that gets displayed (probably) when the EPG files does not exist, rather than exists but has no current events.

That at least would be an easy thing to fix.

Maybe when enough things build up that need fixing I will do a new build, minus the rounds and rounds of beta testings.

Quote:
Many thanks for all the work and effort you have spent writing MyStuff.
You're welcome. and likewise. The best feature of the toppy has always been this community that has supported and developed it. Oh, and the watch one channel while recording two others feature. Oh, and yes obviously the roads.....

_________________
FW: ChunkyWizard Recommended
TAPs:
MyStuff (always one version ahead of everyone else!), and recommended support TAPS
MyStuff skins, manual and latest version: http://www.BobDsMyStuff.co.uk
Known bugs & forthcoming fixes: http://www.BobDsMyStuff.co.uk/Bugs.shtml
Changes coming in the next version: http://www.BobDsMyStuff.co.uk/NextVersion.shtml
View user's profile Send private message Visit poster's website
Geoff Bacon
Posted: Wed Jul 13, 2016 4:39 pm Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
@BobD

I think your post just about nails it.

Let's all hope you get bored when the nights draw in! Having said that, MS is pretty stable so don't spend too much effort.

Of the two problems, I suspect that the first (incorrectly entering recovery mode) is the one that users will see, and panic at, most. It should also be an easy fix.

The second doesn't affect users unless they are in the habit of transferring files or attempt to rebuild (with TopfHDRW?) after corruption. This would probably require some effort to implement especially when considering all the nuances that you have forgotten about (I am a programmer too!)

I would think that caching data for a quicker load is a "bridge too far" although those with large disks would love it.

Cheers
Geoff

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website
Geoff Bacon
Posted: Sun Jul 17, 2016 2:56 pm Reply with quote
Frequent contributor Joined: 12 Jan 2007 Posts: 4559
I've been doing some experiments to try to work out how MS uses the RecordedInfo file.

MS is using the file CDT stamp and the SvcId stored in the recording header to match a recording to the entry in RecordedInfo (changing an existing SvcId results in another record being added to the file)

Deleting a channel, e.g. ITV, results in the archive showing all ITV entries as Channel 4 !!! i.e. it is using the svcNum in the entry to look up the channel name rather than honoring the name in the recording header. It also adds new entries for all the ITV files with a Channel 4 prefix (leaving the original ITV entries still in the file). This means that over time, the file could contain many unwanted records (luckily, the most used channels are at the start of the channel list and are unlikely to be deleted by the user). [But see rescanning - it seems to discard entries]
Note: It doesn't matter whether or not MS is shut down for the channel deletion (either way, it adds the Channel 4 entries)

When adding entries, MS is probably using the SvcNum field to detect the channel name (ITV is normally TV2, Channel 4 is TV3; when ITV is deleted, Channel 4 moves to the TV2 position). It should be matching using the name in the recording header and/or that it in the stored entry (although the channel has been deleted, the stored name is still valid when used to identify the logo to use).

Rescanning, using TM, whilst MS running, to recover the ITV channel results in MS deleting all ITV entries from the file (I'm guessing it removes all entries for newly discovered SvcIds)
The archive still displays these recordings as Channel 4 even though the entries have been removed..
On restarting, MS redetects the ITV recordings and recreates the entries (the archive shows ITV again). These recreated entries are similar but not identical to the original entries (some info has been lost because the original entry included information that was only present in the EPG).

It seems to me that :-
1) The channel name stored in the first field should always match that stored in the recording header i.e. the channel name when the recording was made.
2) When matching entries, MS should be using the Channel name taken from the recording header (rather than matching with the SvcId taken from the recording header)
3) This channel name should be used to find the corresponding logo in logo.dat and the index stored solely in memory to aid future retrieval. i.e. MS should not use the SvcNum to identify the logo.
4) Channels whose names have been changed (i.e. using the same SvcId) would still show an appropriate logo (assuming it is still in the pack). If no logo in the pack then MS could try matching SvcIds to determine whether or not another channel has the same SvcId - if so, it could use it to determine a logo (perhaps cache data about channel name/SvcIds/LogoIndex and look up in cache before attempting to do a big search when a logo can't be found).

I live in hope. Smile

Geoff

_________________
TopManager program
TF5800, TS On, F/W: MS6 Recommended F/W 12/9/2009 -Sy+EvEzPfUUuZ
TAPs: PcControl B1.4; StopExit v1.01; PruneEPG 1.0; fsSave 1.1; QuickJump 1.72; SecCache (UK) v0.4; EIT Sub (Game) v0.6; EPG2MEI v0.96; MyStuff 6.6; Bookmark 3.0; Extend v1.7; Font Manager 1.0d; MyInfo B5.6; MHEG On/Off A3;
Sig generated by EMJB's MyInfo.tap on 29/12/18
View user's profile Send private message Visit poster's website

Display posts from previous:  

All times are GMT + 1 Hour
Page 1 of 1

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum