Kitz Forum

Computer Software => Windows => Topic started by: sheddyian on September 02, 2012, 05:27:41 PM

Title: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 02, 2012, 05:27:41 PM
I wondered if anyone here has much knowledge of Java, especially compiling Java using the JRE.

Background : I use a neat program called Project-X, for editing recordings from my Satellite receiver before burning them to DVD or putting them on my media server computer to watch on the TV later on.

Project-X is ostensibly source-code only, though does come with a pre-compiled version "for the lazy"  ;D  It's this (project-x.jre) I've been using for some time. 

The problem is that, seemingly randomly, when I try to start it, it freezes the whole machine before it's even drawn it's splash screen as it loads.  I have to power off.  It happens occasionally, but often enough to be annoying.

The instructions recommend that you compile your own version of projectX to be tuned to your platform, which seems an excellent idea.  So I downloaded the latest JRK from Oracle, and it compiled without any problems.

All good, except my newly compiled version, which is fractionally larger in size than the original sample, is much much slower. 

Clearly, editing a video recording is going to shift a lot of data about, but the original compiled version would do an hours video in about 5 minutes or less.  My compiled version takes nearer 15 minutes.  It is almost painful to watch it crawl along.

(The original supplied compiled Project-X, and my newly compiled one are the same version, they therefore appear to be from the same source code.  Nothing else has changed, same machine, same Java environment, same configuration parameters for the program)

Any thoughts?

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: kitz on September 02, 2012, 06:51:41 PM
Its a long time since Ive done any Java programming (I used to use the SDK and have no experience of the Oracle version)

>>  you compile your own version of projectX to be tuned to your platform. 

I may be wrong, but I thought the whole idea of Java and the JDK is that it is totally cross-platform!!  Once compiled the code will run on any machine regardless of the o/s.... as its the VM in the JRE which interprets the code.
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: asbokid on September 02, 2012, 07:25:12 PM
Where is that build script? Does it have configuration parameters?  Maybe the compiler options - speed optimisation flags, or chosen platform architecture, perhaps - are not the same?  e.g. pre-built compilation supporting SMP cores, maybe. Whereas your compilation is optimised for retro 386 core, or some such.  Just a guess.

A common view of Java is that it's usually too slow for serious processing tasks like AV work.    The overheads of running Java 'bytecode' in a Java virtual machine on the native CPU cores can be appreciable.

Are there any other tools to do the same job?  If it's understood what you're doing  -  "editing recordings from my satellite receiver before burning to DVD, etc", then maybe avidemux would be good?   I use it on Linux, but it's ported to Windows. It's written in C - blistering fast - the bee's knees.

cheers, a

http://avidemux.berlios.de/download.html
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 02, 2012, 08:13:16 PM
Its a long time since Ive done any Java programming (I used to use the SDK and have no experience of the Oracle version)
Oracle = Sun Java (as was)

Quote
I may be wrong, but I thought the whole idea of Java and the JDK is that it is totally cross-platform!!  Once compiled the code will run on any machine regardless of the o/s.... as its the VM in the JRE which interprets the code.

That's the theory, and indeed it works, as the pre-compiled .JRE runs mostly fine on my XP PC.  Except when it causes the machine to lock  >:(

The release notes suggest local compilation is A Good Thing for performance though.

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 02, 2012, 08:27:41 PM
Where is that build script? Does it have configuration parameters?  Maybe the compiler options - speed optimisation flags, or chosen platform architecture, perhaps - are not the same?  e.g. pre-built compilation supporting SMP cores, maybe. Whereas your compilation is optimised for retro 386 core, or some such.  Just a guess.


Quite possible.  The build script supplied is in a DOS format, which surprised me!  The only configuration parameter was the path to the JRK.  Here's a link to the download :

http://sourceforge.net/projects/project-x/ (http://sourceforge.net/projects/project-x/)

Quote
A common view of Java is that it's usually too slow for serious processing tasks like AV work.    The overheads of running Java 'bytecode' in a Java virtual machine on the native CPU cores can be appreciable.

Are there any other tools to do the same job?  If it's understood what you're doing  -  "editing recordings from my satellite receiver before burning to DVD, etc", then maybe avidemux would be good?   I use it on Linux, but it's ported to Windows. It's written in C - blistering fast - the bee's knees.

I think I wondered about Java speed when I first started using Project-X, but it seemed pretty quick, and my machine is hardly top end (AMD 64 X2 running at 2.2Ghz with 32Bit XP). 

When I was running my much slower compiled version, I had a quick look at resources, it certainly wasn't hogging CPU.  Plenty of idle time on both cores.

I will have a look at AVIDemux.  I suspect I did have a look at it when I first wanted to do the editing, and for a reason that I now forget, I chose Project-X over other solutions.

What I'm trying to do (without crashes) is to take the recorded Transport Stream set various edit points and demux into video and audio stream(s), which I can then remux into something that's broadly DVD compliant without re-encoding.  (I Currently use FFMPEG to remux).  Currently I've only got SD recordings in MPEG2, though at some point I'll get a newer sat receiver that does MPEG4.

(I do know that Project-X doesn't support Mpeg4 or H.262, so it's time is limited anyway unless it get s a major update!)

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: asbokid on September 02, 2012, 09:31:35 PM
Sounds an interesting project!

Just looking at the build script (build.sh) in the CVS.

The build produces its own .jar archive. It appears to compile against two pre-built Java function libraries,  commons-net-1.3.0.jar  and jakarta-oro-2.0.8.jar.

Quote

javac -encoding "ISO-8859-1" -deprecation -O -g:none -classpath lib/commons-net-1.3.0.jar:lib/jakarta-oro-2.0.8.jar -d build @sources.lst


See:
http://commons.apache.org/net/
http://jakarta.apache.org/oro/

Perhaps the problem could be in those pre-built libraries? Maybe try extracting both of them from the pre-built (faster) version of Project-X, and then compile your code against them instead.

Also what is that '-O' option on the javac (java compiler) command line in that build script?

A '-O' option is not even listed with this java compiler (javac 1.6.0_24 from java-6-openjdk).   Conventionally, at least for C compilers, '-O' is normally the Optimisation flag.

Lots of discussion about Javac optimisation (or lack of) on stackoverflow..

http://stackoverflow.com/questions/4997513/javac-optimization-flags

Java presumably has performance profiling tools.  Not personally ever used them.

Good luck  :)

cheers, a
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 02, 2012, 10:30:56 PM
Thanks for that info, Asbokid!

I'm using the Oracle (formerly Sun) Java Development kit, version 1.7.0_07, but on your prompting, I've just checked javac.exe -help and there is no mention of a -O option there either.

I'm also using build.bat, as I'm using Windows  ;D but it looks like it does the same things as build.sh .

Since those pre-built libraries are supplied as part of the Project-X download, would it not be a reasonable assumption that they are the same as the ones the speedy but crashy projectX.jar have been compiled against?

I am hugely rusty in these sort of areas, I've not written a single line of Java and it's probably about 20 years since I last compiled or assembled anything!  So this is on the cusp of being interesting / challenging and being too much grief for little gain. 

That said, I may have a different view in the morning!  I will, at any rate, have a look at AVIDemux. 

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: asbokid on September 02, 2012, 10:50:39 PM
Since those pre-built libraries are supplied as part of the Project-X download, would it not be a reasonable assumption that they are the same as the ones the speedy but crashy projectX.jar have been compiled against?

Very likely.

The code downloaded from the sourceforge project homepage looks quite old.

http://sourceforge.net/projects/project-x/

It compiles okay, but with lots of deprecation warnings..

Code: [Select]
asbo@home:~/Documents/projectx/Project-X_0.91.0$ ./build.sh
src/net/sourceforge/dvb/projectx/common/Common.java:404: warning: [deprecation] stop() in java.lang.Thread has been deprecated
mainprocess.stop();
           ^
src/net/sourceforge/dvb/projectx/common/Resource.java:450: warning: [deprecation] toURL() in java.io.File has been deprecated
return file.toURL();
           ^
src/net/sourceforge/dvb/projectx/xinput/ftp/XInputFileImpl.java:599: warning: [deprecation] readLine() in java.io.DataInputStream has been deprecated
return in.readLine();
         ^
src/net/sourceforge/dvb/projectx/gui/FileProperties.java:391: warning: [deprecation] show() in java.awt.Window has been deprecated
show();
^
src/net/sourceforge/dvb/projectx/gui/CollectionProperties.java:141: warning: [deprecation] show() in java.awt.Window has been deprecated
show();
^
src/net/sourceforge/dvb/projectx/gui/PatchDialog.java:215: warning: [deprecation] show() in java.awt.Dialog has been deprecated
this.show();
    ^
src/net/sourceforge/dvb/projectx/gui/SubpictureFrame.java:312: warning: [deprecation] show() in java.awt.Window has been deprecated
show();
^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:176: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
tfServer.setNextFocusableComponent(tfPort);
        ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:183: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
tfPort.setNextFocusableComponent(tfUser);
      ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:189: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
tfUser.setNextFocusableComponent(tfPassword);
      ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:194: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
tfPassword.setNextFocusableComponent(tfDirectory);
          ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:200: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
tfDirectory.setNextFocusableComponent(testButton);
           ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:206: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
testButton.setNextFocusableComponent(okButton);
          ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:217: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
okButton.setNextFocusableComponent(cancelButton);
        ^
src/net/sourceforge/dvb/projectx/gui/FtpChooser.java:221: warning: [deprecation] setNextFocusableComponent(java.awt.Component) in javax.swing.JComponent has been deprecated
cancelButton.setNextFocusableComponent(tfServer);
            ^
src/net/sourceforge/dvb/projectx/gui/GuiInterfaceImpl.java:74: warning: [deprecation] show() in java.awt.Window has been deprecated
startup.show();
       ^
src/net/sourceforge/dvb/projectx/gui/GuiInterfaceImpl.java:94: warning: [deprecation] show() in java.awt.Window has been deprecated
teletextpagematrix.show();
                  ^
src/net/sourceforge/dvb/projectx/gui/GuiInterfaceImpl.java:138: warning: [deprecation] show() in java.awt.Window has been deprecated
presettings.show();
           ^
src/net/sourceforge/dvb/projectx/gui/GuiInterfaceImpl.java:253: warning: [deprecation] show() in java.awt.Window has been deprecated
processwindow.show();
             ^
src/net/sourceforge/dvb/projectx/gui/GuiInterfaceImpl.java:364: warning: [deprecation] show() in java.awt.Window has been deprecated
CommonGui.getSubpictureFrame().show();
                              ^
src/net/sourceforge/dvb/projectx/gui/ProcessWindow.java:980: warning: [deprecation] show() in java.awt.Window has been deprecated
show();
^
src/net/sourceforge/dvb/projectx/gui/HexViewer.java:506: warning: [deprecation] show() in java.awt.Window has been deprecated
show();
^
src/net/sourceforge/dvb/projectx/gui/MainFrame.java:1537: warning: [deprecation] show() in java.awt.Window has been deprecated
new Html("http://project-x.sourceforge.net/optional/resources/").show();
                                                                ^
src/net/sourceforge/dvb/projectx/gui/MainFrame.java:1649: warning: [deprecation] show() in java.awt.Window has been deprecated
new Html().show();
          ^
src/net/sourceforge/dvb/projectx/gui/MainFrame.java:1853: warning: [deprecation] show() in java.awt.Window has been deprecated
autoload.show();
        ^
src/net/sourceforge/dvb/projectx/gui/MainFrame.java:2294: warning: [deprecation] show() in java.awt.Dialog has been deprecated
ftpChooser.show();
          ^
src/net/sourceforge/dvb/projectx/gui/GOPEditor.java:699: warning: [deprecation] show() in java.awt.Dialog has been deprecated
this.show();
    ^
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
27 warnings
added manifest
adding: remcoll.gif(in = 910) (out= 786)(deflated 13%)
adding: matrix.gif(in = 954) (out= 831)(deflated 12%)
adding: leftcut.gif(in = 918) (out= 788)(deflated 14%)
...
[..etc..]

I just tried both versions  - the pre-built version and the re-compiled version above (both being 0.91.0.00/31.03.2011). And the latter is unstable.

Without intervention, the GUI of the re-compiled version keeps re-sizing the window, and the "used" (memory) field of the Memory Monitor is darting all over the place.

Same java VM, same CPU..

Whereas, the CVS version is apparently 0.91.0.03/31.05.2012.

The resultant jars from re-compiling both versions (0.91.0.00/31.03/2011 and 0.91.0.03/31.05/2012) and the original jar (0.91.0.00) are here, if any use:

https://docs.google.com/folder/d/0B6wW18mYskvBXzFuRUkzZ19mNkU/edit

cheers, a
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 02, 2012, 11:29:54 PM
i'm just trying them now..
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 02, 2012, 11:50:30 PM
hmm

think its an ini file thing  :-[

some stats in a moment...


Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 03, 2012, 12:02:17 AM
Thanks for the two new recompiles Asbokid.  Was interested to see the file size is smaller than mine, made me wonder if they'd be more streamlined!

I did a simple test against them.  I loaded Saturday's Doctor Who, which had about 5 minutes leeway before and after the programme in the original .ts file.  I marked the beggining and end of the programme, and set it processing.  The source .ts file is on an external USB 2 Hard disk, the destination for the demuxed files is internal SATA hard disk.

Each version of ProjectX.jar was run in it's own folder, with the same modified .ini file as the others EXCEPT for the "original setup", which has a much more modified .ini.  And that appears to be the key!

Code: [Select]
version            time mm:ss
Your recompile          03:40
Your CVS recompile      04:02
My recompile            03:51
original setup          01:34
orig .jar in new folder 03:54


I'd taken the supplied .ini file, and copied out of my modified .ini only the bits I thought were relevant... clearly I've left out something that affects performance.  Maybe ProjectX does something by default that I've turned off and it runs faster...  just going to compare the settings and look for something relevant.

Incidentally, all of them ran completely stable, with the display looking fine on my machine. 
I wouldn't totally trust my timings, they seem to vary a bit, and I only ran each variant once, but it was immediately apparent that the original setup was much faster.

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 03, 2012, 12:27:30 AM
Yep, definately the ini file settings.

I copied all the versions of the .jar file into the original folder where it would pick up the original .ini and got the following results after a reboot:

Code: [Select]
CVS version            01:15
my compile             01:12
original               01:10
CVS version try 2      01:13
CVS version try 3 JQS  01:16


(JQS is the Java Quick Starter, which did seem to make ProjectX load faster, but didn't make it run faster, possibly slowing it slightly!)

Tomorrow I shall go through the ini file and see what the differences are.  There are a lot!  Just need to find what's relevant. 

Thanks for your help, sorry it was my ineptitude at not realising the ini could make such a difference!  :-[

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: asbokid on September 03, 2012, 04:12:10 AM
Yeh! Glad you got it sorted Ian!  Do try avidemux, it's really impressive. It's also idiot-proof which is perfect for me!

cheers, a
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 03, 2012, 11:25:47 PM
Thanks for all that Asbokid, and for all the recompiles for me to try.  I have no idea how to work CSV (I imagine it's a distribution system for code updating?) so that was extra helpful.

I'm off to have a bit of a tinker with AVIDemux now :)

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 03, 2012, 11:59:15 PM
um, is there an idiots guide to using AVIDemux anywhere?

Installed Win32 version, runs ok.  Dropped unedited Doctor Who transport stream file onto it as a test.  It wants to re-index this, which takes several minutes. 

It tells me the .ts has multiple audio tracks (which it does) and I must select the main one.  But it doesn't tell me which is which - no sign of PIDs or identification as to which is the main audio and which is the audio description track, and I can't see a way of playing each to find out.  I guess at the first one being main audio.

I marked the start and end of the show with the A and B markers.  Easy  :) Then what do I press to demux?

Added it to the job queue.  Run it.  It complains there are two audio tracks and I need to select the main one.  But I've already done that.  I do it again, add it to the job queue again.  Seems happier this time.  After a few minutes it says its done.  Where has it saved it?  I search and find it in a folder called "jobs".  It has made a file with no extension, so I rename it as .mpg.  Although it's over 1gb in size, it's only 9 minutes long (not the 50 I was expecting) and there's no sound.

I think I've discovered why I chose Project-x - I probably found a dummies guide to using it :)

Any hints you can give me?

Am I wrong in expecting it to spit out 2 x audio tracks in separate files (with file names indicating pids and likely content) and 1 x video file which I can then remux?

Ian
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: asbokid on September 04, 2012, 08:27:31 PM
Hi Ian,

Sorry, missed your message until now.

Not sure about manuals for avidemux. My uses of it have been simple enough that the suck-it-and-see strategy has worked.  Being limited to cutting out unwanted sections footage, splicing several video into one, and transcoding to different formats and different bitrates for upload, DVD and video CD etc.

Once the A and B (start and end points) are set, then just select the output formats (use 'Copy' for both video and audio to stop it from transcoding), and then just 'Save' with a given filename. 

Your other queries are beyond me, sorry.  I'm no AV fiend!

cheers, a
Title: Re: Compiled Java code very slow compared to pre-compiled sample
Post by: sheddyian on September 04, 2012, 08:34:18 PM
HI,

Thanks for the reply, and sorry I was so grumpy about it - I appreciate the time you've taken compiling the .jar s for me and recommending avidemux.  Just got frustrated that it didn't do what I expected.

I did have the video and audio on "copy", but I wasn't clear what sort of container file it was outputting.  If any.  Or why it didn't output the audio as well as the video, separate files or not.

I suspect I'd tried it out in a similar fashion before, and struggled with it, and that's why I moved on to Project-X.  That said, with the lack of HD support in Project-X, it's likely I'll have to move away from it soon, crash or no crash.

Will have another go with Avidemux later on

Thanks

Ian