Stat track app F@H

[Theo]

So I was wondering, if someone in the Linus Tech Tips forums knows enough code or whatever it needs to be able to write a program which can show how many Work Units/ Points a user has contributed to the LTT F@H team. Ive seen this in other forums, so that they track the statistics of each donor and little stuff like that. How many places they gained or lost. Whaler will always be first but yeah you know lol

I feel that this would attract more people from the forum to fold, if for example each week the top donor got a shout out or something like that.

What do you guys think?

[warmon6]

Well, I've tried dipping my hands into programming, Although it hasn't really been my thing (yet). 

 

What I can say is, for anyone that wants stats will need to download these file(s) once per hour to get everything they need.

 

http://folding.stanford.edu/home/teams-stats/

 

 

Monitoring stats updates automatically: available downloads and rules of use

The donor and team stats are updated every hour, although this can be delayed if there were a lot of work units to come back during that hour. We have been turning off web access to the stats database during stats updates (usually on the hour).

 

Please do not use scripts to access the donor or team pages, but use the full donor list (flat files) instead. IP addresses which do not abide by our robots.txt rules will be banned.

 

The donor and team lists are generated now every hour and can be downloaded once an hour from these links:

 

http://fah-web.stanford.edu/daily_user_summary.txt.bz2

http://fah-web.stanford.edu/daily_team_summary.txt.bz2

 

Downloads of the daily donor and team lists should be limited to no more than 24 downloads a day.

 

[Theo]

Well, I've tried dipping my hands into programming, Although it hasn't really been my thing (yet). 

 

What I can say is, for anyone that wants stats will need to download these file(s) once per hour to get everything they need.

 

http://folding.stanford.edu/home/teams-stats/

Thanks, will check it out, but i think something like this would be a cool addition for the folding team!

[warmon6]

Thanks, will check it out, but i think something like this would be a cool addition for the folding team!

 

I should also state that if your wanting to looks at the files (once downloaded), your need a program like 7-zip to extract the file to open it up in notepad or notepad++ (notepad++ recommended).

 

 

 

Now, I'll try to see what I could do to making some stat system but I wouldn't hold your breath on me. 

[alpenwasser]

This site has some pretty detailed stats.

You can look up pretty much anything for anyone from any time.

Also this might be of interest.

Or are you talking about something different?

[warmon6]

Well, I think he's more wanting a stat system that the Linus Folding team made and it's local. 

 

If he was wanting to be led to a site that's already setup, Extreme overclockers stat page is the best there is and already setup to do such.

 

http://folding.extremeoverclocking.com/

 

 

Maybe I'm wrong of what he is wanting but that was my interpretation on this.

[alpenwasser]

Well, I think he's more wanting a stat system that the Linus Folding team made and it's local. 

 

If he was wanting to be led to a site that's already setup, Extreme overclockers stat page is the best there is and already setup to do such.

 

http://folding.extremeoverclocking.com/

 

 

Maybe I'm wrong of what he is wanting but that was my interpretation on this.

Hm, well ok. Seems a bit redundant with all the stats site already out there,

but if somebody wants to code it I won't stand in their way. :D

I personally prefer Free-DC.org because they also include BOINC stats (I've

done both folding and BOINC, so this is a bit more useful for me), but they

seem to have pretty much the same info as EOC when it comes to F@H (donor

performance, threats, opportunities).

[warmon6]

Hm, well ok. Seems a bit redundant with all the stats site already out there,

but if somebody wants to code it I won't stand in their way. :D

I personally prefer Free-DC.org because they also include BOINC stats (I've

done both folding and BOINC, so this is a bit more useful for me), but they

seem to have pretty much the same info as EOC when it comes to F@H (donor

performance, threats, opportunities).

 

 

Well, What ever "floats the boat" i always say.  :) (Meaning, As long as your happy with it, It's fine by me.)

[Mentalguy]

it'd be cool to have it coaded into the forum

[Theo]

it'd be cool to have it coaded into the forum

Yeah thats what i really want! I think it would bring some healthy competition, so more people get into it! Not that anyone would come close to you or whaler but you know!

[L.E.D]

Yeah thats what i really want! I think it would bring some healthy competition, so more people get into it! Not that anyone would come close to you or whaler but you know!

 

I think if some rich guy wants to join he could build an amazing folding farm and beat Whaler.. But the chances of that happening are so low we can really just forget about it xD

[warmon6]

Well, I think I got the first part of this down. Mostly at least. Filtering the F@H flat files.

 

https://docs.google.com/spreadsheet/ccc?key=0AmWO58p5PmPddGFXN1V1TTFnZXJjdWoxTDV1MDU2a3c&usp=sharing

 

Turns out, you can use a spreadsheet program like LibreOffice (what I'm using) or MS Excel to open up the flat files F@H provides (after extracting them of course).

 

Automatically puts everything into columns and rows which makes it far easier for me to filter out those that are not apart of the team.

 

Only 2 down sides though with this method thus far, 

 

1. (at least with LibreOffice) it's takes a very long time to open and process the unorganised/unfiltered files (Takes about 1/2 to 3/4 of an hour to process on my laptop).  

 

(This is all before I start removing the people that isn't apart of the team.)

 

Mainly because there are over 1.6 Million rows of names in the file and LibreOffice is not multi-thread and/or not very well optimize to handle so many rows of data.

 

 

2. LibreOffice has a limit of about 1M rows that can be displayed. So some of the guys are not in my current list.

 

Just an example, EOC say we've had a total of 760 users but I can only see (in total points) the top 691 folders on this team because of that limit.  (At least if you have more than 520 points in total, your on my list. lol)

 

 

Looking past the limitations, I think this is a good starting point. 

 

Next step would be making it automatically update and automate the process. Something I do believe LibreOffice should be able to handle just fine. 

[L.E.D]

Well, I think I got the first part of this down. Mostly at least. Filtering the F@H flat files.

 

https://docs.google.com/spreadsheet/ccc?key=0AmWO58p5PmPddGFXN1V1TTFnZXJjdWoxTDV1MDU2a3c&usp=sharing

 

Turns out, you can use a spreadsheet program like LibreOffice (what I'm using) or MS Excel to open up the flat files F@H provides (after extracting them of course).

 

Automatically puts everything into columns and rows which makes it far easier for me to filter out those that are not apart of the team.

 

Only 2 down sides though with this method thus far, 

 

1. (at least with LibreOffice) it's takes a very long time to open and process the unorganised/unfiltered files (Takes about 1/2 to 3/4 of an hour to process on my laptop).  

 

(This is all before I start removing the people that isn't apart of the team.)

 

Mainly because there are over 1.6 Million rows of names in the file and LibreOffice is not multi-thread and/or not very well optimize to handle so many rows of data.

 

 

2. LibreOffice has a limit of about 1M rows that can be displayed. So some of the guys are not in my current list.

 

Just an example, EOC say we've had a total of 760 users but I can only see (in total points) the top 691 folders on this team because of that limit.  (At least if you have more than 520 points in total, your on my list. lol)

 

 

Looking past the limitations, I think this is a good starting point. 

 

Next step would be making it automatically update and automate the process. Something I do believe LibreOffice should be able to handle just fine. 

 

Great work so far, I am actually really interested in seeing where this project of yours will go.

[warmon6]

Well, just some minor updates.

 

1. I've worked out a way to really speed things up and get around the limitation of LibreOffice Calc not being able to show everyone I need to use in the stats.

 

By using Notepad++, I can filter out most of the users (Libre finish the filtering for anyone else that got though), copy the desired users and paste them in a new text file. That way, I can import everyone on this team into the spreadsheet instead of only grabbing the top 1M world wide users.

 

This also mean the text file I'm bringing into LibreOffice is a lot smaller. Instead of a 1/2 hour+ wait for Libre to open up with 1M+ users, It now opens up in just a few seconds with 700+ users. 

 

 

2. I'm slowly working my way from the back of the line to the front in terms automating the process to have live stats.

 

I just got done with setting up LibreOffice to automatically update the same spreadsheet in google docs with an extension called  OOo2gd.

 https://code.google.com/p/ooo2gd/

 

So any changes that happen on my computer, google docs should be updated to reflect the same thing.

 

 

*I did try the downloadable "google drive" but every time I did a change, I'd had to tell google docs it needed to be opened in spreadsheet and that changed the link every time a change to the spreadsheet was made. This plugin, fixes that issue for me. 

 

here is the current link:

 

https://docs.google.com/spreadsheet/ccc?key=0AmWO58p5PmPddDdwSTJrcHV1eXkzd0dQLVQzLWlaZ1E&usp=sharing

 

Although You wont find to much of a difference as I'm still using the same stat file from this morning (EST) to work on this* 

 

 

 

 

Currently, I need to work on figuring out how to make LibreOffice, Notepad++, 7zip, and the downloading of the flat file to all communicate or at least time correctly to make everything work with each other. 

 

First I have to download the file.

 

Need 7zip to extract the file.

 

Need Notepad++ to reduce the number of users for LibreOffice

 

and last need LibreOffice to organize it all in some fashion.

 

 

 

Once that's all done (how ever I manage to do it), Then I can move on to the next step.

 

 

The next step will be to add stuff to the stats such as PPD, rank changes, daily credit total, ect.  Although that'll be probably a little tricky for me to pull off as most of that stuff will require days worth of stats to be collect before I could effectively use it.

 

 

 

 

 

As for those that have notice I haven't done any actual programming yet, yeah, I'm trying to see how far I can go before having to do such a thing. :P

 

 

Well, I'll update when I have anything worthy enough too say.  B)

[L.E.D]

Well, just some minor updates.

 

1. I've worked out a way to really speed things up and get around the limitation of LibreOffice Calc not being able to show everyone I need to use in the stats.

 

By using Notepad++, I can filter out most of the users (Libre finish the filtering for anyone else that got though), copy the desired users and paste them in a new text file. That way, I can import everyone on this team into the spreadsheet instead of only grabbing the top 1M world wide users.

 

This also mean the text file I'm bringing into LibreOffice is a lot smaller. Instead of a 1/2 hour+ wait for Libre to open up with 1M+ users, It now opens up in just a few seconds with 700+ users. 

 

 

2. I'm slowly working my way from the back of the line to the front in terms automating the process to have live stats.

 

I just got done with setting up LibreOffice to automatically update the same spreadsheet in google docs with an extension called  OOo2gd.

 https://code.google.com/p/ooo2gd/

 

So any changes that happen on my computer, google docs should be updated to reflect the same thing.

 

 

*I did try the downloadable "google drive" but every time I did a change, I'd had to tell google docs it needed to be opened in spreadsheet and that changed the link every time a change to the spreadsheet was made. This plugin, fixes that issue for me. 

 

here is the current link:

 

https://docs.google.com/spreadsheet/ccc?key=0AmWO58p5PmPddDdwSTJrcHV1eXkzd0dQLVQzLWlaZ1E&usp=sharing

 

Although You wont find to much of a difference as I'm still using the same stat file from this morning (EST) to work on this* 

 

 

 

 

Currently, I need to work on figuring out how to make LibreOffice, Notepad++, 7zip, and the downloading of the flat file to all communicate or at least time correctly to make everything work with each other. 

 

First I have to download the file.

 

Need 7zip to extract the file.

 

Need Notepad++ to reduce the number of users for LibreOffice

 

and last need LibreOffice to organize it all in some fashion.

 

 

 

Once that's all done (how ever I manage to do it), Then I can move on to the next step.

 

 

The next step will be to add stuff to the stats such as PPD, rank changes, daily credit total, ect.  Although that'll be probably a little tricky for me to pull off as most of that stuff will require days worth of stats to be collect before I could effectively use it.

 

 

 

 

 

As for those that have notice I haven't done any actual programming yet, yeah, I'm trying to see how far I can go before having to do such a thing. :P

 

 

Well, I'll update when I have anything worthy enough too say.  B)

 

Wow, even more work, great job, I am really surprised that you just picked this up and took off, most times when I hear about someone doing something like a big project such as this they give up. Nice work.

[warmon6]

Wow, even more work, great job, I am really surprised that you just picked this up and took off, most times when I hear about someone doing something like a big project such as this they give up. Nice work.

 

Well, it's sort of a habit of mine to take on new big challenges (that I have dont have a clear idea of how I should do it) and I typically hit them hard.  :lol:  lol

 

 

It's like on my other F@H team (I'm a dual member with this team and Tom's Hardware F@H team), I was asked to build a guide a year ago that people can use and It took me and a few others about a week to complete it. 

 

Here's that guide if you want to look at it.  http://www.tomshardware.com/forum/34683-12-toms-hardware-folding-home-team-guide-testing

 

Maybe there is something you'll spot that might make your F@H client work better for you.  :)

 

Although I need to update it to reflect all the recent changes that have been going on. Certainly with the updated F@H website. 

 

 

 

This is just something that I thought it would be fun to do and I have some spare time to do it.  :D

 

 

 

Although If there are any serious people out there wanting stats and truly know how to program, they can blast anything I do out of the water.   B)

 

 

Now, only if i could attach the spreadsheet directly to the forum....  well technically, there is a way but Linus and Slick dont have it enabled. 

 

If it was, all I have to do is, 

 

Special BBcode button -> Select "Media" -> type in this url for the docs: <iframe width='500' height='300' frameborder='0' src='https://docs.google.com/spreadsheet/pub?key=0AmWO58p5PmPddDdwSTJrcHV1eXkzd0dQLVQzLWlaZ1E&output=html&widget=true'></iframe>

 

and i would have Live stats on the forum in a post/comment. 

 

 

Although I'm certain that they're security reasons for having it disabled (Unless it just something no one asked for).

 

So I'm not going to worry to much. It'll be up to you guys how you'll want to get top the stats and to bug Linus and Slick for that ability or location. 

 

 

Well, time for me to get some sleep. See ya later!

[Whaler_99]

Some BBCode is not enabled for security reasons, others are not enabled simply because we are working on other items in the back end and BBCode is taking somewhat of a back seat :)

 

Once you get the Google Doc cleaned up a bit, we can make a new post with it at the top and I can pin it - least then people can easily find it. Maybe at some point we can also get the BBCode working..

[warmon6]

Understood whaler! 

 

_____________________________________________________________________________________________________________________

 

 

 

As for everyone else, Sorry guys but this project is going to have to take a back seat for the next few days.

 

Combination of the heat in the Washington DC area is just wearing me out (I prefer cold weather over hot) and this weekend, I'll be assisting an American Legion member that's moving to a new house, so I just wont have time to work on this for a little while. 

 

 

Beyond that, I dont have anything new to report (Didn't get the chance to work on it today or yesterday.)

[Whaler_99]

No problem Warmon - thanks for the work up to now. When you do get something you are happy with, let me know an we can pin it... and then you can continue to make changes...

 

As the LTT team becomes more and more popular, hopefully people will jump ship from dead teams, maybe a coder or two.

[warmon6]

Just an update, between yesterday and today, I've been working on the best way on how to automatically download the stats files from F@H.

 

Well, thanks to this site: http://blog.netnerds.net/2007/01/vbscript-download-and-save-a-binary-file/#idc-cover

 

I was able to copy the VBscript code, paste that into Notepad++, input the url file, and Place (and name) the file in a certain location. 

 

 

I can say that a manual test run of the code worked perfectly but i haven't done it automatically just yet. 

 

 

Although I should know if it works around @ 2:10 PM EDT as that is what I set the "Windows Task Scheduler" to run the script.

 

(Waiting till that time to avoid bigger stat sites such as EOC hammering F@H servers and I've already download the same file a few times already in the past hour. So I need to wait to be sure that my IP adress doesn't get block.) 

 

 

 

Once that time come's around, I'll let you know if works or not.

 

 

After this, I'll be working on how to make the different program that i need to organize the file work with each other. 

[warmon6]

Woot, worked like a charm!  B)

 

Now time to work on the next thing.

[warmon6]

Well, knocked out unziping the file using 7-zip.

http://www.dotnetperls.com/7-zip-examples


My .bat script:

@Echo off
cd C:\Program Files (x86)\7-Zip\
7z e C:\Users\(username)\Desktop\stats\Zip\ -oC:\Users\(username)\Desktop\stats\Raw\ -aoa

7z = to launch the program
e = extract the file
-o = output folder (-o has to be touching the output location)
-aoa = Automatically overwrites existing files without prompting.


Maybe by the end of the week I'll have this all done.


Although getting notepad++ to do what I want might slow me down.

(not really seeing command options that I will need to use to open the file, filter the text (whether it's an argument or the ability to call on a macro), and save the new file.)

[Whaler_99]

Looks good @warmon6 - have you looked at textpad? That may have the commend options you need.

[warmon6]

Looks good @warmon6 - have you looked at textpad? That may have the commend options you need.

 

Never heard of it.  :o

 

But a quick glance at it, It does look like it has everything I need.  :D

 

 

Well, time to get back to work on it.

[warmon6]

Well, after looking at the textpad in detail, I notice that it wont be able to fill in all the roles I need. Some of the things I saw when I did a quick glance over it, did not do what I was thinking it would do.

 

 

So what I've learned is that there not really a text editor that can actually do what I want to do. (Then again, I was probably expecting too much for what text editors were really designed to do).

 

 

Although, what I am learning is that there is a way to make a script using VBscript to do all the things I'm needing.

 

I have seen examples that can read .txt files, parse though them, and delete unwanted lines "or" copy desired one onto a new text file

 

 

Only problem that no example, sample, or pre-made script exists that would work with the F@H files in the manner that they're arranged.

 

 

So, it looks like I'll be making my own custom VBscript to do that thing I need.  Guess as the saying goes, "If you want it done right, you have to do it yourself". 

 

 

Down side is, I have no ETA of when I'll be done with the script as I haven't had a need to script in VBscript before. So going to be learning a lot of new things.