Click here to go to the forum index Click here for the home page
 
Author Message

<  TAP and patch development  ~  Simple and fast .REC conversion

Page 1 of 3
Goto page 1, 2, 3  Next
R2-D2
Posted: Mon Jan 08, 2007 5:32 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
Hi,

I've been looking at the Toppy's .rec format and I've found that all I need to do to turn a .rec file into something that is generally usable is to remove the (roughly) 4KB header (to the first 0x47 sync byte). This leaves a normal MPEG Transport Stream (.ts) file which lots of things can play and manipulate.

However, it's desperately inefficient to chop the top off a file since (on most filesystems I've been exposed to) you end up having to basically copy the whole file. So it struck me that a much better idea might be to rewrite the 4KB header part. So I tried inserting 0x47 sync bytes starting from the beginning and at 188 byte intervals (as if it were part of a TS file). This worked perfectly and lined up exactly with the first real 0x47 sync byte. And the file then played perfectly in Media Player Classic (which it didn't if just one of those 0x47 sync bytes was missing).

Now, after the last inserted 0x47 sync byte in the header is invariably a big chunk of zero bytes, which I reckon could be used to store the bytes that had been changed so you could actually change the file back to the original Toppy .rec format just as easily. In fact, in the examples I've looked at there was only one byte that was changed that was variable (i.e. part of the text of the recording description) and couldn't otherwise be easily replaced.

So, it strikes me that it would be fairly easy to write a TAP and/or computer (PC/Mac/Linux) program to do this update (and maybe change the file extension at the same time to .TS to signal the change).

Silly idea or not?
View user's profile Send private message Visit poster's website
nwhitfield
Posted: Mon Jan 08, 2007 5:52 pm Reply with quote
Site Admin Joined: 20 Mar 2005 Posts: 9577
Hmmm; that sounds terribly interesting. Did you just use a text editor, or knock up a tool to do the job?

I shall have to try that with some of the .rec files sitting on the NAS and see if the media player is more amenable to looking at them after that.

Nigel.

_________________
Support this site - make a donation to our running costs
View user's profile Send private message Visit poster's website
R2-D2
Posted: Mon Jan 08, 2007 6:24 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
nwhitfield wrote:
Hmmm; that sounds terribly interesting. Did you just use a text editor, or knock up a tool to do the job?

I used a hex editor called 010Editor, which has the advantage that I could set the width to 188 bytes and have all the sync bytes line up in the first column.
View user's profile Send private message Visit poster's website
bdb
Posted: Mon Jan 08, 2007 6:32 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
Any 'good' codec should be able to cope with an errored bitstream - which is effectively what the .rec header appears as. Sadly, many do get fooled.

rec2ts_v0.02.zip is pc application that will remove the .rec header for you, and create a valid .ts file. It then decodes the .rec header and puts in in a spreadsheet.

Overwriting the .rec header rather than rewriting the entire file would certainly speed up the process; but it doesn't take that long to rewrite a 4GB file on a PC.
If I get a chance, I will add an option to rec2ts to offer this choice.

bdb
View user's profile Send private message
iron
Posted: Mon Jan 08, 2007 8:06 pm Reply with quote
Regular contributor Joined: 03 Jun 2005 Posts: 91 Location: Tunbridge Wells, UK
I would overwrite the rec header with transport stream null packets, i.e. packets with a PID of 0x1FFF. The null packets are used for padding in the transmitted stream and decoders and editors are designed to ignore them.

iron.

_________________
Model: TF5800PVR - Silver Firmware: Recommended_5800; TAPS: UK Timer 1.2, USB Accelerator 1.0, eit_sub, seccacheUK; cables: 10m TF to PC only, 2 metres both ways!
View user's profile Send private message
R2-D2
Posted: Fri Feb 23, 2007 8:43 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
iron wrote:
I would overwrite the rec header with transport stream null packets, i.e. packets with a PID of 0x1FFF.

Ah yes, that would be better. So three bytes per header packet, meaning 60 bytes in all. They still fit in the (as far as I've ever seen) unused last part of the header. I don't know if this is the best place to post this, or if it would be useful to anyone else, but I've converted a quick program I'd written to PHP (for increased portability). This does the quick translation of .REC files to .TS files (including renaming them) -- and back again, too!

Code:

<?php
/* rec2ts.php - Convert Toppy .rec to .ts (and back again) */

/* Rename converted file? */
$dorename = TRUE;

if(count($argv) > 1)
  $filename = $argv[1];
else
  exit("No filename supplied\n");

$handle = @fopen($filename, 'r+b');
if($handle === FALSE)
  exit("Failed to open $filename\n");

fseek($handle,0);
$hdr = fread($handle, 3760);
if($hdr === FALSE)
  exit("Error reading from $filename\n");

$arr = str_split($hdr,188);
if($arr[0][0]==chr(0x47)) {
  foreach($arr as $key => $_) {
    for($i=0; $i < 3; $i++)
      $arr[$key][$i] = $arr[19][$key * 3 + 3 + $i];
  }
  echo "Reverted to Topfield rec file\n";
  $newname = preg_replace('/.ts$/i','.rec',$filename);
} else {
  $repl = array(chr(0x47),chr(0x1F),chr(0xFF));
  foreach($arr as $key => $_) {
    for($i=0; $i < 3; $i++) {
      $arr[19][$key * 3 + 3 + $i] = $arr[$key][$i];
      $arr[$key][$i] = $repl[$i];
    }
  }
  echo "Converted to TS file\n";
  $newname = preg_replace('/.rec$/i','.ts',$filename);
}

// Write the string to the file
fseek($handle,0);
if(fwrite($handle, implode("",$arr)) === FALSE)
  exit("Error writing to $filename\n");

fclose($handle);

// Rename the file if needed
if($dorename && $newname !== $filename)
  rename($filename,$newname);

/* End */
?>
View user's profile Send private message Visit poster's website
matt
Posted: Sat Mar 10, 2007 12:08 am Reply with quote
Forum moderator Joined: 30 Mar 2005 Posts: 2305 Location: St Ives, Cornwall Transmitter: Redruth
I've converted the above script to a droplet for Mac OS X.

Drag and drop a REC or TS file onto it to do a quick conversion back and forth.

notes: only supports one file at a time. doesn't have a fancy icon.

download: http://www.gingerbeardman.com/toppy-rec2ts.zip (58kb)

if you find it useful please let me know.

matt

_________________
PVR: TF5800, MyStuff recommended firmware, new PSU, 250GB Spinpoint HDD, Zalman ZM-OP1 fan TAPs: QuickJump 1.72, MyStuff 6.4, EPG2MEI 0.96, Extend 1.7, Bookmark 3, UK Auto Scheduler 0.73, SecCache 0.4, EIT Sub 0.6, SDS 1.3, TAP Commander 1.34, Font Manager 1.0d. Others: HDFW 2.3a, Channel Manager 5.1, FastScanGUI, MyInfo TV: Philips 37PFL9603D
View user's profile Send private message Visit poster's website
richie
Posted: Tue Jun 03, 2008 8:39 am Reply with quote
Frequent contributor Joined: 30 Mar 2005 Posts: 156 Location: Ash Vale (Hannington)
Out of interest, has anyone ported this into tap format, for use with recordings that a user knows are only to be transferred off the toppy? ie It gets the rec files ready as ts files.

I ask because my XP machine is very short of disk space and rec2ts on win makes a (large) copy of the file first before deleting the original!
View user's profile Send private message
R2-D2
Posted: Tue Jun 03, 2008 8:58 am Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
richie wrote:
Out of interest, has anyone ported this into tap format
There's little point in doing it on the Toppy since the .ts is not playable. To use the script on a PC all you need is PHP (the .exe and .dll as a minimal installation).

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
DB1
Posted: Tue Jun 03, 2008 7:04 pm Reply with quote
Frequent contributor Joined: 30 Mar 2005 Posts: 728 Location: Orpington
Err, why don't you use a player that understands .rec? Like Mplayer.

_________________
TF5800, F/W: 5.13.65AbBfBqC0CbCeCkCwCyDEcEeEfErEsEvEzFFsGmHHeIKtNfOtPPcPePsRRaRhRpRsSSdSrStT2TdTfTpUUuXXpXwXl TAPs: TF5000 Display v1.53; QuickJump 1.72; Power Manager v2.2; MyInfo B5.6; DescriptionExtender 2.23; Remote Extender 1.6; Archivev1.0a; mei2archive BETA 3.8l7; EPGnavigator v5.1c; UK Auto Scheduler v0.73.1; Extend v1.7; Power Restore V0.7.8
Tx: CP
Sig generated by MyInfo on 14/4/13
View user's profile Send private message
richie
Posted: Tue Jun 03, 2008 9:28 pm Reply with quote
Frequent contributor Joined: 30 Mar 2005 Posts: 156 Location: Ash Vale (Hannington)
It's for conversion into divx for my portable media player. I do stream using vlc when I want to watch stuff on my pc.
View user's profile Send private message
richie
Posted: Tue Jun 03, 2008 9:32 pm Reply with quote
Frequent contributor Joined: 30 Mar 2005 Posts: 156 Location: Ash Vale (Hannington)
So to clarify, for a 2Gb rec file, I need 4gb space.

Because rec2ts creates a copy before deleting the original, before something then converts it from ts into mpg. Though I have in the past used mencoder, which can convert it from rec into divx, I am now playing with mp4 scripts on my asus, and the only one i can find available as an ipkg is ffmpeg. ffmpeg is also faster than mencoder.

Whereas if there was a tap which could convert a file I knew I wanted to be on avi only, ie never watch on my toppy, then I would only need about 2.3 Gb on my pc - 2 Gb fo the ts plus 300mb or so for the avi being created.

This is a very custom application!

Perhaps I should just buy a new bigger laptop drive!
View user's profile Send private message
R2-D2
Posted: Tue Jun 03, 2008 9:48 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
richie wrote:
the only one i can find available as an ipkg is ffmpeg.
I expect you can run PHP on your device.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
captain747480
Posted: Mon Jun 16, 2008 5:57 pm Reply with quote
Frequent contributor Joined: 25 Jun 2005 Posts: 499 Location: London Transmitter: Crystal Palace
R2-D2 wrote:
iron wrote:
I would overwrite the rec header with transport stream null packets, i.e. packets with a PID of 0x1FFF.

Ah yes, that would be better. So three bytes per header packet, meaning 60 bytes in all. They still fit in the (as far as I've ever seen) unused last part of the header. I don't know if this is the best place to post this, or if it would be useful to anyone else, but I've converted a quick program I'd written to PHP (for increased portability). This does the quick translation of .REC files to .TS files (including renaming them) -- and back again, too!

Code:

<?php
/* rec2ts.php - Convert Toppy .rec to .ts (and back again) */

/* Rename converted file? */
$dorename = TRUE;

if(count($argv) > 1)
  $filename = $argv[1];
else
  exit("No filename supplied\n");

$handle = @fopen($filename, 'r+b');
if($handle === FALSE)
  exit("Failed to open $filename\n");

fseek($handle,0);
$hdr = fread($handle, 3760);
if($hdr === FALSE)
  exit("Error reading from $filename\n");

$arr = str_split($hdr,188);
if($arr[0][0]==chr(0x47)) {
  foreach($arr as $key => $_) {
    for($i=0; $i < 3; $i++)
      $arr[$key][$i] = $arr[19][$key * 3 + 3 + $i];
  }
  echo "Reverted to Topfield rec file\n";
  $newname = preg_replace('/.ts$/i','.rec',$filename);
} else {
  $repl = array(chr(0x47),chr(0x1F),chr(0xFF));
  foreach($arr as $key => $_) {
    for($i=0; $i < 3; $i++) {
      $arr[19][$key * 3 + 3 + $i] = $arr[$key][$i];
      $arr[$key][$i] = $repl[$i];
    }
  }
  echo "Converted to TS file\n";
  $newname = preg_replace('/.rec$/i','.ts',$filename);
}

// Write the string to the file
fseek($handle,0);
if(fwrite($handle, implode("",$arr)) === FALSE)
  exit("Error writing to $filename\n");

fclose($handle);

// Rename the file if needed
if($dorename && $newname !== $filename)
  rename($filename,$newname);

/* End */
?>
What's the syntax for running this?

I've tried

/opt/bin/php rec2ts.php filename.rec

but I get 'Failed to open filename.rec'

_________________
TF5800 HD103SI 1Tb HDD, TS On,F/W: MS6 Recommended F/W 12/9/2009 -Sy+Pe
TAPs: MyStuff 6.4; TF5000 Display v1.53; MHEG Control A2f; EPG2MEI v0.96; TAP Commander 1.0.3; MyInfo B5.5; AccurateBM v0.3; SecCache (UK) v0.4; Font Manager 1.0d
Other Kit:LG 37LE5900 TV, Zotac ND-22 HTPC running XMBC, Sky+ HD, Panasonic HT-520 Home Cinema, Harmony 885, Asus WL-500GX with Oleg's, 2Tb NAS (Mirrored), Flirc USB Keyboard Emulator
View user's profile Send private message MSN Messenger
R2-D2
Posted: Mon Jun 16, 2008 6:26 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
captain747480 wrote:
I've tried

/opt/bin/php rec2ts.php filename.rec

That looks good to me, although is your recording really called "filename.rec"? And there are no paths mentioned, except on the php command, so is it actually in the same directory as the script?

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website

Display posts from previous:  

All times are GMT + 1 Hour
Page 1 of 3
Goto page 1, 2, 3  Next

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum