Chopping database records

I have been working on a project recently for a client which required the user to insert a record into an existing set of records and to modify those existing records to accomodate the new record.

The above diagrams shows what needed to be accomplished. If you imagine from left to right is time on the diagram, each record has a start date and and end date and the new record is needing to be inserted between those start dates and end dates.

The start and end dates can overlap the start and end dates of the new record and this is where the existing need to be chopped and modified to accomodate the new record.

So how did I go about this?

Firstly I decided to bucket the existing records into categories as follows …

And like …

This gives us four actual categories and a fifth “Ignore” category. The four actual categories are LeftRight, Middle, Left, Right. Some more examples are shown below …

The bucketing of the existing records into categories from a coding perspective is accomplished as follows …

Firstly I selected the existing records, pseudo code shown below …

dateStart = new record start date
dateEnd = new record end date
dbDateStart = existing record start date
dbDateEnd = existing record end date

Other columns could be added to this selection code if needed, in some cases a “user_id” or other foreign key could be potentially needed.

Using the above selection code will select all the existing records that exist within the range of the new record start and end dates. In other words the records that fall into the “Ignore” category will not be selected.

This is why the “Ignore” category is not a real category as the records are never selected and thus do not need to be bucketed.

Now I have selected my records from the database I put the records into a loop. Whilst in this loop I bucket the records into a set of arrays using the following conditions …

LeftRight …

Middle …

Left …

Right …

Now that the records have been bucketed into the arrays I loop through each group and process the groups seperately as follows …

new record = the new record you are wanting to insert into the DB via displacing the current records
original records = the record in the LeftRight bucket that has not been cloned

LeftRight …

I take a clone of the record and I modify the original (none cloned) records date end to be the new record date start -1 day this is then saved. I then take the clone and set the date start to be the newly inserted records date end +1 day this is then saved.

Middle …

Any middle records are deleted.

Left …

I modify the records date end to be the newly inserted records date start -1 day this is then saved.

Right …

I modify the records date start to be the newly inserted records date end +1 day this is then saved.

After all of the bucketed records have been processed there should now be room to insert the new record so I insert / save the new record.

So that concludes how I created a system that would allow a new record to be inserted whilst chopping the existing records to accomodate the new records.

Internal system – Toolkit

I have been using a web based product called ActiveCollab for sometime now …

https://activecollab.com

I have been using ActiveCollab since version 4. When ActiveCollab 5 was introduced it had a complete redesign, less features and a more slick user interface.

More features have been introduced to ActiveCollab 5 though since it was first released.

ActiveCollab is a project management system that also handles Time Tracking and Invoicing plus more.

I will continue using ActiveCollab for projects / task management however for other areas of my business I have created my own internal tool which I have named “Toolkit”.

I built my own system because I wanted a system that would do the following …

Estimate earnings
Fixed fee and retainer projects are entered into the Toolkit. The Toolkit can then estimate monthly and yearly earnings based on the projects entered. The projects also includes a way for me to manage all the projects I am working on.

Incomings
The Toolkit has an area in which all incomings can be entered. The incomings can then be exported in a format appropriate for my accountant.

Outgoings
The Toolkit has an area in which all outgoings can be entered. The outgoings can then be exported in a format appropriate for my accountant.

Mileage
The Toolkit has an area in which all mileages can be entered. The mileages can then be exported in a format appropriate for my accountant.

Leads
The Toolkit has an area in which leads can be managed. I actually pulled this area out of the Neptune – Lead Tracker – http://www.neptuneleadtracker.com software system.

CRM
The Toolkit includes an area in which all the companies and people I have worked for can be managed. ActiveCollab does actually include this feature but the Toolkit version allows more advanced search features and has fields on the create / update forms to match my exact needs.

Timetracking
The Toolkit includes an area in which all time entries can be tracked. ActiveCollab does have this feature.

Invoicing
The Toolkit includes an area in which invoices can be created. Invoices can also be generated from time entries. ActiveCollab does include this feature but the Toolkit version matches my exact needs precisely.

Notes
The Toolkit includes an area to keep notes. ActiveCollab does include this feature.

Files
The Toolkit includes an area to upload files. ActiveCollab does include this feature.

ActiveCollab has a more slick, beautiful user interface but in terms of functionality built specifically for myself obviously the Toolkit wins.

The Toolkit is primarily made up of CRUD screens, no fancy UI / Ajax heavy screens here. Just datagrids and buttons.

I am in the process of live testing the Toolkit I have built and will be moving various functions of my business over to the Toolkit.

King of Adem – progress update – building queue

Progress has been made on the King of Adem project. For those of you not familiar with this project please read the original King of Adem post at …

http://www.jamesbarnsley.com/site/2017/08/13/king-of-adem/

Since the update I have been primarily working on the building queue. The building queue is the queue which buildings get inserted into when a player wishes to construct a building in one of his / her regions.

To get to the construct building screen the user has to be in one of his / her regions so those areas to get to that screen had to be part built.

The areas part built are namely user, game, region and building. Once on the construct building screen the user can select from a list of buildings that can be constructed.

Each building is validated on a case by case basis and if it does not pass the validations the building cannot be constructed, here are some of the validations so far …

Check fund available
The player must have the correct amount of fund available to construct the building. The amount of funds available is the players start of turn funds subtracting any funds that have been used in the current turn.

For example player may have started with 1000 coins, if a player constructs a building for 300 coins then the program gets the players start of turn funds but substracts 300 coins because it knows a building worth 300 coins was requested to be constructed this current turn.

Check building queue full
The building queue may only allow so many buildings to be in the queue at any one time for each region. Currently this number is set at 5 and the number is stored in a config file.

Check building already built
Check if the building has already been constructed in the region. Each building has a “constructionLimitPerRegion” variable which stores the number of times that particular building may be constructed in each region.

Check dependent buildings constructed
Each building can have what are called dependent buildings. These are buildings which need to be constructed first before you can build the current building.

Also the validation knows whether a building is an upgrade of an existing building or a completely different building that just happens to depend on another building being constructed.

All kinds of checking had to be implemented for this validation, whether building is an upgrade / dependant or just dependant, whether advanced tier buildings already exist for lower dependant buildings, whether lower dependant buildings already exist etc.

Check advanced tier building exists
Check if an advanced tier building already exists for the current building wanting to be constructed. If the advanced tier building already exists then there is no reason to build the current building.

Again this needs to take into account that some buildings can be constructed more than once so in some instances it may need to allow the building to be constructed even though the lower building has already been constructed.

The above validations are run everytime a user issues the command to insert a building into the building queue for a particular region. These validations are run from the “Building” Model. Each building is also given its own Model, the hierachy is like as follows …

\yii\db\ActiveRecord -> \app\components\ActiveRecord -> \app\models\Building -> \app\models\BuildingTownBarracks

Or …

\yii\db\ActiveRecord -> \app\components\ActiveRecord -> \app\models\Building -> \app\models\BuildingPort

etc.

I can run special validations for specific buildings by placing those validations in the specific building class. Those validations will only run for the specific building and not all buildings.

That is as far as I have got so far. Still a long, long way to go, more updates to come soon.

King of Adem

The latest personal project I am working on which also happens to be the largest personal project I have worked on so far.

King of Adem is a game for tabletop minature wargamers. Players manage their kingdom through the app. Kingdoms consist of a number of a territories, each player starts with one.

In each territory there is a capital city in which buildings can be constructed. Buildings provide benefits like more money or the ability to recruit armies.

When 2 armies collide by moving into each other the players can then resolve the battle as normal by playing their normal game of tabletop minatures.

The game takes place on an island called Adem. Adem is a recently discovered island where each player is fighting for control. You fight to become the “King of Adem” by conquering the whole island.

Although Adem is stated as “recently” discovered, this only means “recently” in the grand scheme of things. Adem actually has a number of Kings / Factions already on the island and each Kingdom has some infrastructure.

The project is currently divided into 3 parts …

King of Adem Diagram

Part 1 – Game Server
The Game Server is currently in construction and is being developed using the Yii 2 framework written in PHP. The database is MySQL. The server takes view requests and command requests from the client application.

View requests provide data to the client in JSON format. This data has everything needed to render views for the game for example, building, army, territory data.

Command requests provide data to the server about interactions happening in the client application. For example move commands, construct building commands etc.

The decision to use View / Command requests was chosen instead of opening the server up entirely.

Essentially one option could have been for each Data model on the server to have a view, insert, update, delete request to use for either commands, views or both.

This would give the client more freedom at the cost of putting more logic onto the client.

As the first client is being developed using IOS and IOS uses a view per screen type of structure then View requests make sense. Giving the client the exact data the client needs in order to render the view.

Part 2 – Game Test Client
The Game Test Client is currently being developed alongside the Game Server and serves as a testing ground for the development of the server.

The Game Test Client can send View requests to the server and read / present the data from the server in a user friendly way. The Game Test Client can send command requests to the server.

The Game Test Client is being developed using the Yii 2 framework written in PHP. The Game Test Client is my own personal client for testing the server and is not the actual client the end user / player will be using.

Part 3 – IOS Game Client
The IOS Game Client is the actual client the end user / player will be using to play King of Adem. The IOS Game Client as you might have guessed will be an IPhone application. There may also be the ability to run the application on the IPad as well.

I decided to develop the IOS Game Client because I wanted the game to be portable and it cannot get much more portable than having the game on your phone.

I also choose the game to be an IOS application because I want to improve my IOS skills and offer IOS development as part of the everyday services I offer to my clients.

Once these 3 parts are complete the game will be playable and released to the public. Part 1 and 2 are already in construction, Part 3 will be constructed after parts 1 and 2 and completed.

After the game has been released to the public, a number of other client applications could potentially be developed including an Android version and a full game client developed in Unity3D or similar.

The IOS client will be more management like as opposed to a graphical game. The IOS application could include graphics etc, but it will not be at the level of a full game like experience. This is why I suggest a later developed Unity3D or similar game engine development.

I will be keeping everyone informed of the development of this game through the blog so keep checking back for further updates.

Yii / Yii2 checking controller params at the access control level

One of the areas I notice in a lot of Yii / Yii 2 web applications is the checking of parameter variables in what I consider the wrong areas.

Here is an example of one of my controller methods for saving a Book …

So what stands out about this method? No load Model method? No checking to see whether the Book has a valid ID? How will we know whether the Book has a valid ID, how will we know whether the Book is suppose to be accessed by the particular user that is logged into the system?

The answer is, the access rules and bizrules of the web software application. All controller methods should be assigned to an access rule. If the controller method contains parameters then the controller method should be assigned to an access rule as well as a bizrule.

For smaller / less complex applications you can actually emulate the bizrule by using “matchCallback” which is spoken about in this article I wrote previously …

http://www.jamesbarnsley.com/site/2017/06/13/yii-2-user-permission-using-matchcallback/

You can also read my previous article on what I think of the Yii / Yii2 load Model method …

http://www.jamesbarnsley.com/site/2015/04/22/yii-loadmodel-method/

All controller parameters should be checked and validated in the bizrule, by the time the parameter reaches the controller method there should no need to have any checking in the controller method at all for the controller parameters.

If the Book does not exist or the Book is not accessible by the current logged in user then that will be handled before any controller method code gets executed in the first place.

There is no reason to have this level of checking in the controller method itself. Doing it the way I have described also keeps the controller more clean as extra code is not needed to check the controller parameters.

Solution for using DatePickers on forms in Yii2

Sometimes in web software applications we need to collect dates such as a start date or end date using forms. In this article I will be using the Kartik DatePicker widget which can be found here …

http://demos.krajee.com/widget-details/datepicker

For the purposes of this article a master Model will need to be created. I discussed how to create master Models and Controllers in the following article …

Create a master Model and Controller in Yii2

So based on the master Model now being created the article can continue. Add the field to the form using code similar to this. Include the plugin in the form …

Add the field to the form …

In the master Model 2 methods needed to be created …

And …

These methods can be called to save the “date” data to the database from any of the child Models. The best place to call the methods would be in the “beforeSave” and “afterFind” methods as follows …

And …

Depending on how you have setup the validations for the “date” attribute you may need to run the “saveDateFormat” and “dateFormat” methods in the “beforeValidate” and “afterValidate” methods as well as the “beforeSave” and “afterFind”.

This depends on whether your validations are formatting the date as the date would be saved to the database or validating against the format the date appears on the form.

If your validations validate against the format the date saves to the database then calling “saveDateFormat” and “dateFormat” methods in the “beforeValidate” and “afterValidate” methods will not be needed.

I generally setup my validations so I do not need to run the “saveDateFormat” and “dateFormat” methods in the “beforeValidate” and “afterValidate” methods only through the “beforeSave” and “afterFind” methods.

Using the above code you can implement multiple datepickers on the same form or across different forms. Follow the same logic, add the field or fields to the form and call the “saveDateFormat” and “dateFormat” methods from the child Model.

Yii 2 User Permission using matchCallback

As a web software developer I sometimes need to implement some kind of user permissions in the web applications that I build. The Yii 2 framework which I currently use has a built in system for doing just this however for applications that need a more lightweight user permission system the Yii 2 framework has introduced a method which was not available in the Yii 1 framework that can be used to easily build lightweight user permissions and that method is “matchCallback”.

Both Yii 1 and Yii 2 have a user permissions system named RBAC. In Yii 1 RBAC is implemented by writing code to define the user permissions, this code then populates the database tables with the user permissions. The user permissions are then applied to controller methods via more code known as access rules.

User permissions can also have what is known as a “bizRule”, a “bizRule” is an additional piece of code executed with the user permission that needs to evaluate to true in order for the user permission to be true. This can be used to implement checks to see if the record being displayed on the page is owned by the user trying to access the page as well as various other checks. The “bizRule” code is stored in the database along with the user permission.

One of the differences between the Yii 2 implementation of RBAC and the Yii 1 version is that the Yii 2 “bizRule” code is stored in code files and not the database. This makes more sense to me as the “bizRule” code is kept alongside the other code in the frameworks, models, controllers, views etc.

Infact it makes so much sense that during the later stages of using the Yii 1 framework I actually just stored class / function references in the “bizRule” code and wrote my real “bizRule” alongside the rest of framework code. Essentially implementing my own version of what Yii 2 covers as standard.

As mentioned above Yii 2 now implements a method called “matchCallback”. This means for web applications requiring simple user permissions RBAC is not needed as the “matchCallback” can be used to define the “bizRule” directly in the access rules …

So as shown above for web applications only requiring simple user permissions, the access rules are all that are needed.

array_map plus other possibilities

When building web applications certain patterns appear in our code. A pattern that use to appear in my code quite regularly is shown in the example below …

Essentially we have an array of data and we want to create a new array out of only part of the original data. The above code can be abstracted using the PHP array_map function as shown in the example below …

PHP includes a number of these functions within its library. PHP also gives you the ability to create your own. The above array_map function can be created in PHP as follows …

Using code similar to the above is also the key to implementing iteration functions on collections allowing you to write code as such …

Or filtering …

The possibilities this opens are numerous. If you would like to know more about writing code similar to the above then I can recommend the following book …

https://adamwathan.me/refactoring-to-collections/

Refactoring to Collections – The Definitive Guide to Curing the Common Loop by Adam Wathan

Select2 library solution in Yii2 part 4 – retrieving the data

To continue on from the previous article I wrote about saving the Select2 data …

http://www.jamesbarnsley.com/site/2016/11/07/select2-library-solution-in-yii2-part-3-saving-the-data/

I will now talk about how to the retrieve the stored Select2 data so that the data can be displayed on the front-end Select2 field. Following on from the Task / Countries example, place the following “loadSelect2Items” method inside the Master model or Task model if you did not create a Master model …

The code above can be explained as follows …

  1. Line 3 – return the array of Select2 data using the “array_map” function
  2. Line 4 – store the related entity models name into a variable, in this case that would be the Country models name. This uses the Country relationship that was created in “Part 2” of this article series
  3. Line 5 – store the IDs of the related relationship entity model in the return data, in this case that would be the IDs of the Country model records
  4. Line 6 – find all the records for the model / model property in the “select2_data” table, in this case that would be the Task model and “countries” property

The next step is use the “loadSelect2Items” in the Task model to the load the Select2 data into the “countries” property, place the following inside the “afterFind” method of the Task model …

So that concludes the Select2 library solution for Yii2. I have covered the creation of the necessary database table and the read / write methods for reading / writing the data.

After the initial methods have been setup this solution will provide you with a way of easily putting a Select2 field on your forms that allows for many to many relationships between models with just a couple of line of code.

Select2 library solution in Yii2 part 3 – saving the data

To continue from the previous article …

http://www.jamesbarnsley.com/site/2016/10/31/select2-library-solution-in-yii2-part-2-database/

Once the database table has been created the code can now be written to display the Select2 field on the form and save the Select2 data to the database table.

In this article I will demonstrate the Select2 functionality as though we are wanting to add “Tags” to a Task model. When I say “Tags” I mean the standard web term for Tags. Essentially these are words that you can tag to an entity for example a WordPress article that categorizes the WordPress article using the tags allowing a user to search by those words.

When I say Task model, this is a fictional model in a Yii2 application that will allow the user to create and update tasks. For the purposes of this article it is assumed you will create your own Task model along with the controller and views. Create a standard CRUD for tasks possibly giving each task a name and a description.

You will need to create a Tag database table to store the Tag data along with its associated model. Create a database table with “id” field for the primary key and a “name” field to store the Tag name. Create a standard Yii2 model for the Tag table, this model does not need any methods inside it, it is a blank Yii2 model that extends ActiveRecord.

After you have created your task CRUD functionality and the Tag database table with associated model the Select2 functionality can be integrated.

First the field is added to the form using the Kartik Select2 plugin. The plugin can be found here …

http://demos.krajee.com/widget-details/select2

Add the field to the form using code similar to this. Include the plugin in the form …

Add the field to the form …

Create a “tags” property in the Task model, this will be used to store the tags field data on both saving and retrieving …

Now the field is added to the form and you have a property in which to store the array of data which the Select2 library needs in order to function. Next you need to use the “saveSelect2Items” items method as shown below to save the array data in the “tags” property to the database.

The “saveSelect2Items” method can be placed inside the Task model but it would be better in some kind of global area such as a Master model as is it likely that you will want to use the functionality in more than one model / form. I wrote an article on how to create Master models which can be read here …

http://www.jamesbarnsley.com/site/2016/03/14/create-a-master-model-and-controller-in-yii2/

The “saveSelect2Items” method is as follows …

The code above can be explained as follows …

  1. Line 3 – get the class name of the model object in this case that would be the Task model and convert it to lowercase characters
  2. Line 4 – get the class name of the store model object in this case that would be the Tag model and convert it to lowercase characters
  3. Line 5 – get the Select2 array data of the model object in this case that would be the “tags” property in the Task model
  4. Line 7 – if the model object Select2 property is not empty, in this case that would be the “tag” property
  5. Line 9 – get the current stored records in relation to the model and model properties we are using. In this case the Task model, Tag model and “tag” property
  6. Line 11 – loop through the currently stored records
  7. Line 13 – if the related model object ID is not in the model object property array data. In this case that is if the Tag ID is not in the “tag” property array data
  8. Line 15 – set the deleted flag to 1, which indicates deleted
  9. Line 16 – save the record
  10. Line 22 – loop through the array data in the model object property, in this case that is the Task model “tags” property
  11. Line 24 – check to see whether a record already exists in relation to the model and model properties we are using. In this case the Task model, Tag model and “tag” property
  12. Line 26 – if no existing record can be found
  13. Line 28 to 33 – create a new Select2 record and set the properties
  14. Line 35 and 36 – save the new Select2 record but if it does not save then return false
  15. Line 42 – the else part of the original if statement, if there is no array data in the “tags” property
  16. Line 44 – delete all the existing records for the model we are dealing with. In this case that is the Task model

As mentioned in the database article the “saveSelect2Items” method soft deletes the unwanted Select2 records with a little adaptation this could be altered to hard delete the records.

Now that the “saveSelect2Items” has been created you can call it in your Task model by placing the following code in the “beforeSave” method of the Task model …

The first parameter of the “saveSelect2Items” is the property used to store the Select2 array data, the second parameter is the original model that uses the Select2 field on the form in this case that is the Task model. The third parameter is the relation model in this case that is the Tag model.

The code up to this point puts a Select2 field on a form and saves the Select2 data when the form is saved. If you go back to the form the Select2 field will not be populated with the saved data yet as that is covered in the next article which talks about retrieving the Select2 data.