Real Time Data Collecting – ver2

In the last post I explained how I collect the data using Azure by means of Webjobs, event hubs etc, there is a simpler version. Python.

I’m still using Azure but this time I have a VM which has a series of Python scripts. These scripts are run at different times based on the requirements. The actual Architecture diagram looks like this.

Basically, I use Windows scheduler to run different Python scripts at various times.

  • 5 minutes interval – These are used for up-to-date transfers of fantasy premier League players. The scripts extract the information from the json files and inserts it into a SQL db. My on prem SQL db runs a stored procedure  at a predefined time which moves the data across then deletes the old records from the cloud. This enables me to keep the db in Azure at a decent size.
  • Weekly interval – These are used to get the data based on points, minutes played, cards received etc. The new data is pulled across to an on prem db in the same way as the transfer data.

The Python scripts works in the following way.

  • A main script is run via the schedule, this calls the specific Python files that reads the data. The DB connections are held in another Python file. This way I have one connection file which the others ‘IMPORTS’

We are at the tail end of the season, so all these scripts will be sued in ‘Anger’ next season. I have been using them mind you and I have found some good insights.


My team for wk34 which is a double game week I used some of this data to pick my players.

  • Captain – Eriksen: 2 games and he has been doing well recently. So far he has scored 1 goal, so I should expect a decent return.
  • GK – Schmeichel: 2 games so I took out WBA keeper for him, my other choice was Pickford so against Man Utd I would have never thought they would keep a clean sheet. Still 1 game to go for Kasper so I might get something. His points were slightly swayed by his penalty save.

My other players were my stalwarts which have been in there for a long time, mind you the City players didn’t return much, now they are champions maybe time for a change.

It’s an interesting time of the season for a few reasons.

  • Premiership race is over, so it’s the next three places that are up for grabs, so Spurs, Liverpool, Man Utd and Chelsea players have it all to play for.
  • Relegation is still not decided so teams will be fighting for their Premiership survival and expect unusual results Like Man Utd getting beat at Home by WBA.

I need to look at the fixtures and the stats together next. Next season this will be an automated process. I’m hoping to click on a button and it will give me a narrative on the coming weeks 🙂

Loads more to do really, the thing is the more data I look at the more data I want. Its like going round in circles 🙂