Altering PostgreSQL columns from one enum to another

Altering the type of a column in postgres from one enum type to another requires an intermediate cast to  text. Additionally, if there is a default value for the field, it cannot be altered and instead must be dropped and re-added as the new type.

Simple case: New type contains all old-type values

In the simple version, the new enum type contains (at least) all of the same labels as the old one. For instance, as with these two:

No default on column

If a table has an old_enum column and we want to turn it into a new_enum one, with no default value in place on the column, we can use the following command:

The USING expression casts the current value of column_name to text, and then to new_enum. This works because every allowed value of the first enum type exists in the second.

With a default value

This case is not significantly more difficult to deal with. If there is a default value on the column, we simply remove it before altering the enum type of the column, and then add a new one when we’re done:

Converting enum labels

A more complicated scenario arises when not all of the old labels appear in the new enum. I’ll assume that there is a mapping from the old ones to the new, at least for every label that is known to appear in a row of the table. If there isn’t, then the conversion is probably not a good idea in the first place.

Consider now an even newer type,

We still want to convert from the old_enum type, but now we also want to map the label ‘a’ to ‘alpha’, and ‘b’ to ‘beta’, while leaving ‘c’ and ‘d’ alone. This can be accomplished by manually applying each required change via a CASE  statement in the USING  expression:

For each record, this statement returns ‘alpha’ or ‘beta’ if the column contains ‘a’ or ‘b’, respectively, or otherwise returns the current value in the column cast as text. The returned value in all cases is then cast to the newer enum type.

Arrays of joined ids

Thanks to a helpful stack overflow response, I now have an improved solution to my problem of needing to add arrays of joined ids to JSON output from CakePHP 3. The output from a test containing Widget models that can each have many Foos looks something like this:


To easily and automatically add this capability to any model I wanted, I created a new class extending Cake’s  Table , which I called “ApiTable”:

The  formatWithJoinedIds() method adds an array called  model_ids for each model, from an input list, that is associated with this one through a ‘BelongsToMany’ or ‘HasMany’ join, without including the contents of the associated models. It is intended to be called from the  findApi() method, which simply performs a query containing the passed-in desired list of models and formats the results.

The custom finder is straightforward:

The array  $includeIdsFor should contain strings identifying the associated properties whose joined ids should be included. For instance, if a Widget has many associated Foos, Bars, and Bazzes, the ids of the foos and bars would be included in the output by passing  ['Foos', 'Bars'].

The result formatter, used to add the calculated fields containing the id arrays, is a little longer but not much more complicated:

This routine generates an array of all properties of the current model that point to many of another model. It then iterates over these properties, and for each one iterates over all of the foreign models to collect an array of their ids. The resulting array is added as a new property to the row, and the foreign models themselves are unset if they are not to be included in the output.

Listing associated IDs during a Controller query in CakePHP 3

Yesterday I looked into adding a key-value pair such as  "tag_ids": [1, 2, 3] to a JSON object returned by a serialized CakePHP view, for Bookmarks belonging to many Tags. My solution produces the desired result, but involves the execution of a new database query for every bookmark, on top of the one that retrieved the record in the first place. This is not exactly desirable behaviour, so today I looked into other options. I still don’t have what I consider to be an optimal solution, but did come up with an alternative that manages to pull all the associated Tag ids in the same query that retrieves the current batch of Bookmarks.

There is a new disadvantage introduced by this solution, because the resulting KVP contains a string instead of an array, looking like  "tag_ids": "[1, 2, 3]" . Additionally, the code is nowhere near ready to be generalized for easy addition to multiple controllers, nor even particularly elegant in terms of making use of Cake’s routines for inflection etc. Nevertheless, I want to record it while it’s still fresh in my mind.

The idea is to use the CakePHP query builder to LEFT JOIN the Bookmarks model’s table to the join table that contains its many-to-many relationships with the Tags model. The query is then grouped by all of the Bookmarks fields, and the  tag_id field from the join table is aggregated into a single comma-delimited string. I’m using PostgreSQL, so accomplish this with the string_agg command. Here’s a working example:

I found that I had to set autofields(false)  to avoid having CakePHP automatically include every field in the join table, and therefore needing to add them to the group-by clause or aggregating them in some fashion.

There are a few improvements that can be made right away. Additional joins can be added, but will introduce multiples in the concatenated string unless the string aggregation is made distinct,  string_agg(distinct Book...). Bookmarks with no tags yield   "tag_ids": null, but can be made to give "tag_ids": "[]"  by wrapping the aggregation in coalesce(..., '').

Larger-scale improvements could involve outputting an array instead of a string in the JSON, perhaps by returning a postgres array instead of a CSV string, and teaching Cake how to deal with that properly. Beyond that, not hard-coding the table names, generating the list of joins automatically, and generally wrapping this all up into a behaviour or trait would be nice steps to take.

Array of associated IDs in CakePHP 3.0 “belongsToMany” relationship

Today I was struggling with how to get CakePHP to return a JSON representation of a model, including a simple array of the foreign-key ids from the join table that specifies a mutual belongsToMany relationship (formerly, hasAndBelongsToMany or HABTM). For a concrete example, I wanted to build on the Bookmarker tutorial by creating an API endpoint to retrieve bookmarks, each containing an array of its tag ids. Something like this:

Using Cake’s data views via the RequestHandler and _serialize elements made serving the JSON straightforward enough for the Bookmark model without the tags. Adding the tags to the output was easy enough using  contain() to retrieve associated data. This lead to having the entire tag included in the result though, not the compact “tag_ids” array I had in mind. Even selecting only the id field and setting autofields(false) left an array of objects, including extraneous join information. Instead of containing integers, the tags array of each bookmark contained objects that looked like this,

where a simple  1 was all I wanted.

To solve this problem, I ended up using a virtual field on the Bookmark model that creates the desired array of ids, and which can be easily serialized to JSON.

First, as with other approaches to the data view, the RequestHandler had to be added to either the Bookmarks controller or the App controller.

Next add the virtual tag_ids field through the magic method _getTagIds(), which queries the join table Bookmarks_Tags to select the tag_id for every tag associated with the current bookmark_id. This list is then used to populate a standard PHP array of the integer ids, which becomes the value of the virtual field.

Then all it took in the Bookmarks controller was to query for the additional non-virtual fields to be included, and store the results in a serialized variable: