Chopping database records

I have been working on a project recently for a client which required the user to insert a record into an existing set of records and to modify those existing records to accomodate the new record.

The above diagrams shows what needed to be accomplished. If you imagine from left to right is time on the diagram, each record has a start date and and end date and the new record is needing to be inserted between those start dates and end dates.

The start and end dates can overlap the start and end dates of the new record and this is where the existing need to be chopped and modified to accomodate the new record.

So how did I go about this?

Firstly I decided to bucket the existing records into categories as follows …

And like …

This gives us four actual categories and a fifth “Ignore” category. The four actual categories are LeftRight, Middle, Left, Right. Some more examples are shown below …

The bucketing of the existing records into categories from a coding perspective is accomplished as follows …

Firstly I selected the existing records, pseudo code shown below …

dateStart = new record start date
dateEnd = new record end date
dbDateStart = existing record start date
dbDateEnd = existing record end date

Other columns could be added to this selection code if needed, in some cases a “user_id” or other foreign key could be potentially needed.

Using the above selection code will select all the existing records that exist within the range of the new record start and end dates. In other words the records that fall into the “Ignore” category will not be selected.

This is why the “Ignore” category is not a real category as the records are never selected and thus do not need to be bucketed.

Now I have selected my records from the database I put the records into a loop. Whilst in this loop I bucket the records into a set of arrays using the following conditions …

LeftRight …

Middle …

Left …

Right …

Now that the records have been bucketed into the arrays I loop through each group and process the groups seperately as follows …

new record = the new record you are wanting to insert into the DB via displacing the current records
original records = the record in the LeftRight bucket that has not been cloned

LeftRight …

I take a clone of the record and I modify the original (none cloned) records date end to be the new record date start -1 day this is then saved. I then take the clone and set the date start to be the newly inserted records date end +1 day this is then saved.

Middle …

Any middle records are deleted.

Left …

I modify the records date end to be the newly inserted records date start -1 day this is then saved.

Right …

I modify the records date start to be the newly inserted records date end +1 day this is then saved.

After all of the bucketed records have been processed there should now be room to insert the new record so I insert / save the new record.

So that concludes how I created a system that would allow a new record to be inserted whilst chopping the existing records to accomodate the new records.

Yii / Yii2 checking controller params at the access control level

One of the areas I notice in a lot of Yii / Yii 2 web applications is the checking of parameter variables in what I consider the wrong areas.

Here is an example of one of my controller methods for saving a Book …

So what stands out about this method? No load Model method? No checking to see whether the Book has a valid ID? How will we know whether the Book has a valid ID, how will we know whether the Book is suppose to be accessed by the particular user that is logged into the system?

The answer is, the access rules and bizrules of the web software application. All controller methods should be assigned to an access rule. If the controller method contains parameters then the controller method should be assigned to an access rule as well as a bizrule.

For smaller / less complex applications you can actually emulate the bizrule by using “matchCallback” which is spoken about in this article I wrote previously …

http://www.jamesbarnsley.com/site/2017/06/13/yii-2-user-permission-using-matchcallback/

You can also read my previous article on what I think of the Yii / Yii2 load Model method …

http://www.jamesbarnsley.com/site/2015/04/22/yii-loadmodel-method/

All controller parameters should be checked and validated in the bizrule, by the time the parameter reaches the controller method there should no need to have any checking in the controller method at all for the controller parameters.

If the Book does not exist or the Book is not accessible by the current logged in user then that will be handled before any controller method code gets executed in the first place.

There is no reason to have this level of checking in the controller method itself. Doing it the way I have described also keeps the controller more clean as extra code is not needed to check the controller parameters.

Solution for using DatePickers on forms in Yii2

Sometimes in web software applications we need to collect dates such as a start date or end date using forms. In this article I will be using the Kartik DatePicker widget which can be found here …

http://demos.krajee.com/widget-details/datepicker

For the purposes of this article a master Model will need to be created. I discussed how to create master Models and Controllers in the following article …

Create a master Model and Controller in Yii2

So based on the master Model now being created the article can continue. Add the field to the form using code similar to this. Include the plugin in the form …

Add the field to the form …

In the master Model 2 methods needed to be created …

And …

These methods can be called to save the “date” data to the database from any of the child Models. The best place to call the methods would be in the “beforeSave” and “afterFind” methods as follows …

And …

Depending on how you have setup the validations for the “date” attribute you may need to run the “saveDateFormat” and “dateFormat” methods in the “beforeValidate” and “afterValidate” methods as well as the “beforeSave” and “afterFind”.

This depends on whether your validations are formatting the date as the date would be saved to the database or validating against the format the date appears on the form.

If your validations validate against the format the date saves to the database then calling “saveDateFormat” and “dateFormat” methods in the “beforeValidate” and “afterValidate” methods will not be needed.

I generally setup my validations so I do not need to run the “saveDateFormat” and “dateFormat” methods in the “beforeValidate” and “afterValidate” methods only through the “beforeSave” and “afterFind” methods.

Using the above code you can implement multiple datepickers on the same form or across different forms. Follow the same logic, add the field or fields to the form and call the “saveDateFormat” and “dateFormat” methods from the child Model.

Yii 2 User Permission using matchCallback

As a web software developer I sometimes need to implement some kind of user permissions in the web applications that I build. The Yii 2 framework which I currently use has a built in system for doing just this however for applications that need a more lightweight user permission system the Yii 2 framework has introduced a method which was not available in the Yii 1 framework that can be used to easily build lightweight user permissions and that method is “matchCallback”.

Both Yii 1 and Yii 2 have a user permissions system named RBAC. In Yii 1 RBAC is implemented by writing code to define the user permissions, this code then populates the database tables with the user permissions. The user permissions are then applied to controller methods via more code known as access rules.

User permissions can also have what is known as a “bizRule”, a “bizRule” is an additional piece of code executed with the user permission that needs to evaluate to true in order for the user permission to be true. This can be used to implement checks to see if the record being displayed on the page is owned by the user trying to access the page as well as various other checks. The “bizRule” code is stored in the database along with the user permission.

One of the differences between the Yii 2 implementation of RBAC and the Yii 1 version is that the Yii 2 “bizRule” code is stored in code files and not the database. This makes more sense to me as the “bizRule” code is kept alongside the other code in the frameworks, models, controllers, views etc.

Infact it makes so much sense that during the later stages of using the Yii 1 framework I actually just stored class / function references in the “bizRule” code and wrote my real “bizRule” alongside the rest of framework code. Essentially implementing my own version of what Yii 2 covers as standard.

As mentioned above Yii 2 now implements a method called “matchCallback”. This means for web applications requiring simple user permissions RBAC is not needed as the “matchCallback” can be used to define the “bizRule” directly in the access rules …

So as shown above for web applications only requiring simple user permissions, the access rules are all that are needed.

array_map plus other possibilities

When building web applications certain patterns appear in our code. A pattern that use to appear in my code quite regularly is shown in the example below …

Essentially we have an array of data and we want to create a new array out of only part of the original data. The above code can be abstracted using the PHP array_map function as shown in the example below …

PHP includes a number of these functions within its library. PHP also gives you the ability to create your own. The above array_map function can be created in PHP as follows …

Using code similar to the above is also the key to implementing iteration functions on collections allowing you to write code as such …

Or filtering …

The possibilities this opens are numerous. If you would like to know more about writing code similar to the above then I can recommend the following book …

https://adamwathan.me/refactoring-to-collections/

Refactoring to Collections – The Definitive Guide to Curing the Common Loop by Adam Wathan

Select2 library solution in Yii2 part 4 – retrieving the data

To continue on from the previous article I wrote about saving the Select2 data …

http://www.jamesbarnsley.com/site/2016/11/07/select2-library-solution-in-yii2-part-3-saving-the-data/

I will now talk about how to the retrieve the stored Select2 data so that the data can be displayed on the front-end Select2 field. Following on from the Task / Countries example, place the following “loadSelect2Items” method inside the Master model or Task model if you did not create a Master model …

The code above can be explained as follows …

  1. Line 3 – return the array of Select2 data using the “array_map” function
  2. Line 4 – store the related entity models name into a variable, in this case that would be the Country models name. This uses the Country relationship that was created in “Part 2” of this article series
  3. Line 5 – store the IDs of the related relationship entity model in the return data, in this case that would be the IDs of the Country model records
  4. Line 6 – find all the records for the model / model property in the “select2_data” table, in this case that would be the Task model and “countries” property

The next step is use the “loadSelect2Items” in the Task model to the load the Select2 data into the “countries” property, place the following inside the “afterFind” method of the Task model …

So that concludes the Select2 library solution for Yii2. I have covered the creation of the necessary database table and the read / write methods for reading / writing the data.

After the initial methods have been setup this solution will provide you with a way of easily putting a Select2 field on your forms that allows for many to many relationships between models with just a couple of line of code.

Select2 library solution in Yii2 part 3 – saving the data

To continue from the previous article …

http://www.jamesbarnsley.com/site/2016/10/31/select2-library-solution-in-yii2-part-2-database/

Once the database table has been created the code can now be written to display the Select2 field on the form and save the Select2 data to the database table.

In this article I will demonstrate the Select2 functionality as though we are wanting to add “Tags” to a Task model. When I say “Tags” I mean the standard web term for Tags. Essentially these are words that you can tag to an entity for example a WordPress article that categorizes the WordPress article using the tags allowing a user to search by those words.

When I say Task model, this is a fictional model in a Yii2 application that will allow the user to create and update tasks. For the purposes of this article it is assumed you will create your own Task model along with the controller and views. Create a standard CRUD for tasks possibly giving each task a name and a description.

You will need to create a Tag database table to store the Tag data along with its associated model. Create a database table with “id” field for the primary key and a “name” field to store the Tag name. Create a standard Yii2 model for the Tag table, this model does not need any methods inside it, it is a blank Yii2 model that extends ActiveRecord.

After you have created your task CRUD functionality and the Tag database table with associated model the Select2 functionality can be integrated.

First the field is added to the form using the Kartik Select2 plugin. The plugin can be found here …

http://demos.krajee.com/widget-details/select2

Add the field to the form using code similar to this. Include the plugin in the form …

Add the field to the form …

Create a “tags” property in the Task model, this will be used to store the tags field data on both saving and retrieving …

Now the field is added to the form and you have a property in which to store the array of data which the Select2 library needs in order to function. Next you need to use the “saveSelect2Items” items method as shown below to save the array data in the “tags” property to the database.

The “saveSelect2Items” method can be placed inside the Task model but it would be better in some kind of global area such as a Master model as is it likely that you will want to use the functionality in more than one model / form. I wrote an article on how to create Master models which can be read here …

http://www.jamesbarnsley.com/site/2016/03/14/create-a-master-model-and-controller-in-yii2/

The “saveSelect2Items” method is as follows …

The code above can be explained as follows …

  1. Line 3 – get the class name of the model object in this case that would be the Task model and convert it to lowercase characters
  2. Line 4 – get the class name of the store model object in this case that would be the Tag model and convert it to lowercase characters
  3. Line 5 – get the Select2 array data of the model object in this case that would be the “tags” property in the Task model
  4. Line 7 – if the model object Select2 property is not empty, in this case that would be the “tag” property
  5. Line 9 – get the current stored records in relation to the model and model properties we are using. In this case the Task model, Tag model and “tag” property
  6. Line 11 – loop through the currently stored records
  7. Line 13 – if the related model object ID is not in the model object property array data. In this case that is if the Tag ID is not in the “tag” property array data
  8. Line 15 – set the deleted flag to 1, which indicates deleted
  9. Line 16 – save the record
  10. Line 22 – loop through the array data in the model object property, in this case that is the Task model “tags” property
  11. Line 24 – check to see whether a record already exists in relation to the model and model properties we are using. In this case the Task model, Tag model and “tag” property
  12. Line 26 – if no existing record can be found
  13. Line 28 to 33 – create a new Select2 record and set the properties
  14. Line 35 and 36 – save the new Select2 record but if it does not save then return false
  15. Line 42 – the else part of the original if statement, if there is no array data in the “tags” property
  16. Line 44 – delete all the existing records for the model we are dealing with. In this case that is the Task model

As mentioned in the database article the “saveSelect2Items” method soft deletes the unwanted Select2 records with a little adaptation this could be altered to hard delete the records.

Now that the “saveSelect2Items” has been created you can call it in your Task model by placing the following code in the “beforeSave” method of the Task model …

The first parameter of the “saveSelect2Items” is the property used to store the Select2 array data, the second parameter is the original model that uses the Select2 field on the form in this case that is the Task model. The third parameter is the relation model in this case that is the Tag model.

The code up to this point puts a Select2 field on a form and saves the Select2 data when the form is saved. If you go back to the form the Select2 field will not be populated with the saved data yet as that is covered in the next article which talks about retrieving the Select2 data.

Select2 library solution in Yii2 part 2 – database

To continue from the previous article …

http://www.jamesbarnsley.com/site/2016/10/24/select2-library-solution-in-yii2-part-1-introduction/

I will now talk about the database table needed to get my implementation of the Select2 library working. As mentioned earlier the original design for the database included a table for each set of Select2 relations. So if I wanted a Person to be able to select multiple Countries I would have a Select2 PersonCountries table, 1 table per relation.

The latest design for the database has just 1 table to store all of the Select2 relations used throughout the web software application. I find this easier as I can then use the Select2 like a module, whenever I want Select2 in my web software application I just drop in the code files along with the database table.

The database table I am using is as follows …

I will now run through what the columns in the database table are used for …

  1. id – unique integer for the record
  2. entityModel – the short name of the model class stored in lowercase characters, for example the “Person” model
  3. entityModelField – the name of the model property used on the form to select the related Select2 relations, for example “countries”
  4. entityModelID – the ID of the model, for example the ID of the “Person” model
  5. relationEntityModel – the short name of the related model class in lowercase characters, for example the “Country” model
  6. relationEntityModelID – the ID of the related model, for example the ID of the “Country” model
  7. deleted – a flag to determine if this record is deleted
  8. createdBy – the ID of the user this record was created by
  9. createdDate – the date this record was created
  10. updatedBy – the ID of the user who last updated this record
  11. updatedDate – the date this record was last updated

Please note that “id” is just a standard primary key and is not unique to this article also deleted, createdBy, createdDate, updatedBy and updatedDate are not unique to this article. These are just standard columns that I populate all of my tables with and are not really necessary for the purposes of this article.

The deleted column may be necessary if you are running the code exactly as explained in this article but the record could have just as easily have been hard deleted and hence the deleted column is just personal preference.

To know more about the standard columns I use and to auto populate these columns read the following article …

http://www.jamesbarnsley.com/site/2016/04/04/populating-standard-columns-in-a-database-table-using-yii-2/

The above database table is the table I use to power the Select2 library relations in my web software applications. An example use case for this can be described as follows …

You have a update Person form (entityModel) with a countries field (entityModelField) that allows the user to select various Countries that are related to the person (the ID of the Person is the entityModelID). You can select various Countries and save them to the Person. For each country saved a select2_library record will be created, the ID of the Country is the relationEntityModelID and the Country model itself is the relationEntityModel.

At this point a Select2Library model will need to be created for the “select2_library” table. In the example used in this article a relationship will need to be created between the Select2Library model and the Country model. This will need to be created using the “relationEntityModelID” like this …

All of the above will become clearer when actually running the code and seeing the results. The code to get this to run will be described in the follow up articles.

Select2 library solution in Yii2 part 1 – introduction

For those of you who do not know the Select2 library is a Javascript / JQuery replacement for select boxes. Although there are a few options to choose from in terms of using the Select2 field the configuration I will be looking at will be the multi select configuration.

The multi select configuration is useful for allowing the user to select things that require multiple choices, a typical example of this could be Tags but at a deeper level the multi select configuration is useful for allowing the user to create many to many relationships between models on forms.

An example Select2 field is shown below, allowing the selection of countries on a form …

Select2 Library

The above Select2 field could be on a add / edit person form allowing the many to many relation between People and Countries.

The solution I am providing in this article is all about saving and retrieving the Select2 data in a way that will streamline the process so a Select2 field can be attached to any of your forms within your web software application and it will just work out of the box.

The front-end part of the Select2 field is solved already as you can use the following Yii2 plugin for that …

http://demos.krajee.com/widget-details/select2

The above plugin displays the Select2 field on your Yii2 web forms. The above plugin passes the Select2 field data in the POST as an array. It is what to do with that data in a consistent and streamlined way that this article I am writing covers.

There are many ways the data could be handled for example the data could be JSON encoded and stored in a column in the database table then JSON decoded and displayed on the form when retrieving. Another way would be to have relationship tables for each relationship between models and store the data that way.

The way I have chosen is more like the later except that I do not have relationship tables for each relationship, I simply have 1 central Select2 relationship table that handles all Select2 relationships across the whole web software application.

What is the purpose of all this?

Most programmers would just use the Krajee Select2 plugin (link above) and leave it at that in terms of abstraction. The programmer would manually write the save / retrieve code specific to each instance of the Select2 plugin most likely in the Controller. The way I am using the Select2 plugin is to have methods that handle all of this.

If I want to use the Select2 plugin field in any of my forms, with just a few method calls the Select2 plugin field will be up and running, saving / retrieving data for any field / model. The way I am doing this also integrates better with Yii2 and makes use of the Yii2 relations to fetch associated Select2 data, so this way integrates deeper with Yii2.

Allowing the user to select items on a form and have the software application save those items as a many to many relationship is not always an easy task. This method takes away some of the complexity of that, so that I can have user defined many to many relationships between models / database tables.

Generating a unique random string for Model properties in Yii2

When building web software applications I sometimes need a way to generate unique random strings. Unique random strings can be used as character keys or tokens to identify database records without having to give away the database records ID number. I personally use unique random strings to produce clickable links in my “Request lost password” emails. The unique random string can be matched to a user record and the password can be re-created and sent to the user.

In this article I will show you a method of creating unique random strings. Yii2 already has a way of generating random strings but these will not always necessarily be unique, so in cases where they need to be, this method will come in use. The method I am about to show is named “generateUniqueRandomString”. I would recommend placing this method in a master Model as it is likely that you are going to want to use this method in more than 1 Model and want it exposed to many different Models.

I covered how to create master Models in a previous article which you can read here …

Create a master Model and Controller in Yii2

After you have created the master Model make sure you put the following “use” statement within it …

Now place the following “generateUniqueRandomString” method inside the master Model …

The above code can be explained as follows …

  1. Line 3 – Use the standard Yii2 “generateRandomString” method to generate a random string of characters
  2. Line 5 – IF statement to check whether the random string already exists for the Model, checked against the attribute (or DB column) passed in through the methods parameters
  3. Line 6 – If the random string does not exist already then return the random string
  4. Line 8 – If the random string does exist already, then recursively run the method again and try and generate a different random string that does not already exist

You can use the above method like follows …

The above code shows a book being found and a unique random string being generated for the books “characterKeyString” property. The book is then saved.

The length by default is set to 32 which means it will generate a 32 character unique random string. If you want to change the length just pass in the second parameter as the length you want to generate.

This method is very useful for when it is important that a random string is unique. I have personally used this method in my user creation, lost password request and file storage parts of my web software applications.