PostgreSQL La base de donnees la plus sophistiquee au monde.

La planete francophone de PostgreSQL

mercredi 24 mai 2017

Philippe Florent

Partitionnement avec PostgreSQL 10

Partitionnement et sous-partitionnement combinés avec indexation BRIN et parallélisme : un cocktail efficace

mercredi 24 mai 2017 à 21h30

mardi 23 mai 2017

Loxodata

Monitoring avec Postgres

Voici un rapide retour sur la solution de Steve Simpson pour le monitoring d’infrastructures complexes donné le jour du PGDAY Paris, le 23 mars dernier. Les slides sont disponibles sur slideshare.

Le contexte : une infrastructure complexe avec de gros volumes de données et de charges (Multi-tenant massively parallel workload).

La problématique : comment monitorer tout cela ?

Deux réponses possibles

La première : ELK (Elastic Search - Logstash - Kibana) associé à InfluxDB + Grafana + SQLite avec en sus Kafka, MySQL et Zookeeper. Et tout cela fait beaucoup d’outils à mettre en place comme vous pouvez-le voir ci-dessous : Un monitoring classique (illustration tirée des slides de Steven Simpson)

Ou alors… Simplement PostgreSQL ! (+Grafana) La simplification du monitoring avec PostgreSQL (illustration tirée des slides de Steven Simpson)

Steve commence par passer en revue les outils existants, avec leur entrée sur le marché. Les besoins sont listés un à un, les fonctionnalités employées une à une et ce, pour chacune des briques de l’infrastructure. L’intégralité des outils employés au départ est remplacée par PostgreSQL (et Grafana pour la partie graphique).

Prenons les choses dans l’ordre

Il est en effet possible de faire absorber à PostgreSQL un gros volume d’information avec des structures de schema dénormalisées comportant des champs de type JSON et JSONB qui permettent une grande flexibilité de contenu.

Les différentes métriques sont stockées de cette façon ainsi que la liste des métriques elle même.

Les possibilités du language SQL dans PostgreSQL sont mises à contribution (fonctions TIME_ROUND, AVG, BETWEEN, @>…), de même que les index GIN (sur les champs JSONB).

Les fonctions utilisateur sont par la suite employées sur des parties de schéma normalisées pour diminuer la redondance des informations, organiser les données et les métriques brutes absorbées.

La pertinence des élements à observer

Pour grapher les données, une granularité minimale est nécessaire pour obtenir une représentation pertinente. Le filtrage/échantillonage et la renormalisation du schéma permettent de faciliter le requêtage et de diminuer la quantité de données en transit.

Pour les résumés (aggregats de données), ce sont encore les fonctions internes (TIME_ROUND, LEAST, GREATEST…) qui sont mises à profit ainsi que les triggers mais également les vues (jointures).
Ces résumés permettent d’obtenir un état général toutes les 5 minutes. Ce jeu de données utilisé est donc faible et l’impact sur les performances est encore une fois limité par une utilisation judicieuse des points forts du SGBD.

Une image vaut mieux qu’un long discours

La conférence est accompagnée de graphiques comparatifs de performance sur les temps de requêtages. L’orateur nous donne des exemples de requêtes, fonctions, triggers qui nous permettent de nous rendre compte de ce qu’il a fallu mettre en place en terme de language SQL.

Les insertions de données dans les différents régimes (dénormalisé, normalisé, résumé) sont comparées. Insertion des données de monitoring dans PostgreSQL (illustration tirée des slides de Steven Simpson)

On s’aperçoit que les 600ms, 1750ms et 3300ms nécessaires pour absorber 45 millions de lignes (le monitoring d’une journée) sont des performances très intéressantes. Ce qui fait de PostgreSQL un sérieux challenger dans la liste des outils en compétition sur le marché des solutions de monitoring.

L’orateur nous propose les évolutions qu’il entrevoit pour son système. Notamment des résumés sur 30 minutes avec des requêtes plus larges. Du partitionnement par jour pour faciliter l’ingestion et la suppression régulière sur la durée. Un processing par lignes en batch avec des intervales. Et une ingestion asynchrone de l’ingestion des transactions.

Steve nous explique qu’il cherche à historiser sur 6 mois avec des espaces disques inférieur au Terabyte. Une autre contrainte imposée est de n’avoir que 100ms de requête.

Tirer au mieux partie des fonctionnalités présentes

Mais ça n’est pas tout, les autres fonctionnalités utilisées incluent la recherche dans les logs (type rsyslog) en stockage clé/valeur sur des champs JSONB (index GIN) avec FULL TEXT SEARCH (ts_query/ts_vector).

Pour finir, il utilise le parsing de log avec les REGEX contre une liste de patterns (stockés en tables pour plus de flexibilité). Les informations ainsi extraites sont insérées dans une table de logs avec un champ JSONB. Au vu des performances lors de l’ingestion de données pour les différents types de champs, Steve propose d’utiliser PostgreSQL pour le queuing également.

Tous les outils cités au début de la conférence sont finalement remplacés. Les informations brutes absorbées, puis ordonnées ne nécessite maintenant plus qu’une interface ergonomique pour être exploitées. Grafana est alors utilisé pour cette partie. Pour Steve, PostgreSQL est une boite à outils de données persistantes qui, de surcroit, utilise en natif du SQL.

Les utilisateurs de PostgreSQL ne cesseront jamais de m’étonner. Tant de flexibilité et de fonctionnalités !

par contact@loxodata.com (Loxodata) le mardi 23 mai 2017 à 15h30

dimanche 21 mai 2017

Philippe Florent

Indexation BRIN

Mise à jour des 3 articles sur l'indexation Block Range (BRIN) avec la version 10 beta de PostgreSQL

dimanche 21 mai 2017 à 13h00

jeudi 18 mai 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 7 mai 2017

Le PGDay Austin 2017 aura lieu le samedi 26 août. L'appel à conférenciers s'éteint le 4 juin : https://pgdayaustin2017.postgresql.us

[ndt: MeetUp à Nantes le mardi 20 juin : https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Nantes/]

[ndt: MeetUp à Paris le jeudi 29 juin : https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Paris/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170507211724.GA31062@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Robert Haas pushed:

Peter Eisentraut pushed:

  • Fix logical replication launcher wake up and reset. After the logical replication launcher was told to wake up at commit (for example, by a CREATE SUBSCRIPTION command), the flag to wake up was not reset, so it would be woken up at every following commit as well. So fix that by resetting the flag. Also, we don't need to wake up anything if the transaction was rolled back. Just reset the flag in that case. Author: Masahiko Sawada <sawada.mshk@gmail.com> Reported-by: Fujii Masao <masao.fujii@gmail.com> https://git.postgresql.org/pg/commitdiff/9414e41ea703ea5fcc288bcf7dc000e53306896b
  • Don't wake up logical replication launcher unnecessarily. In CREATE SUBSCRIPTION, only wake up the launcher when the subscription is enabled. Author: Fujii Masao <masao.fujii@gmail.com> https://git.postgresql.org/pg/commitdiff/a99448ab4515aaadc17647e53633f418893f5adf
  • doc: Update ALTER SEQUENCE claims about changes being nontransactional. Clarify that all changes except RESTART are transactional (since 1753b1b027035029c2a2a1649065762fafbf63f3). Reported-by: Michael Paquier <michael.paquier@gmail.com> https://git.postgresql.org/pg/commitdiff/a35ac7c4e3ccf93876b4652d94a418fc82e0eda3
  • Avoid unnecessary catalog updates in ALTER SEQUENCE. ALTER SEQUENCE can do nontransactional changes to the sequence (RESTART clause) and transactional updates to the pg_sequence catalog (most other clauses). When just calling RESTART, the code would still needlessly do a catalog update without any changes. This would entangle that operation in the concurrency issues of a catalog update (causing either locking or concurrency errors, depending on how that issue is to be resolved). Fix by keeping track during options parsing whether a catalog update is needed, and skip it if not. Reported-by: Jason Petersen <jason@citusdata.com> https://git.postgresql.org/pg/commitdiff/3d092fe5409b98272ddd6e623b657308a3c5f004
  • doc: Add missing markup. https://git.postgresql.org/pg/commitdiff/460c89f46c1fdf11baa8e76e6d04e1ff87d7e008
  • doc: Improve order in ALTER PUBLICATION/SUBSCRIPTION ref pages. Move the OWNER and RENAME clauses to the end, so the interesting functionality is listed first. This is more typical on nearby reference pages, whereas the previous order was the order in which the clauses were added. https://git.postgresql.org/pg/commitdiff/e9500240661c03750923e6f539bfa2d75cfaa32a
  • Fix cursor_to_xml in tableforest false mode. It only produced <row> elements but no wrapping <table> element. By contrast, cursor_to_xmlschema produced a schema that is now correct but did not previously match the XML data produced by cursor_to_xml. In passing, also fix a minor misunderstanding about moving cursors in the tests related to this. Reported-by: filip@jirsak.org Based-on-patch-by: Thomas Munro <thomas.munro@enterprisedb.com> https://git.postgresql.org/pg/commitdiff/0de791ed760614991e7cb8a78fddd6874ea6919d
  • Prevent panic during shutdown checkpoint. When the checkpointer writes the shutdown checkpoint, it checks afterwards whether any WAL has been written since it started and throws a PANIC if so. At that point, only walsenders are still active, so one might think this could not happen, but walsenders can also generate WAL, for instance in BASE_BACKUP and certain variants of CREATE_REPLICATION_SLOT. So they can trigger this panic if such a command is run while the shutdown checkpoint is being written. To fix this, divide the walsender shutdown into two phases. First, the postmaster sends a SIGUSR2 signal to all walsenders. The walsenders then put themselves into the "stopping" state. In this state, they reject any new commands. (For simplicity, we reject all new commands, so that in the future we do not have to track meticulously which commands might generate WAL.) The checkpointer waits for all walsenders to reach this state before proceeding with the shutdown checkpoint. After the shutdown checkpoint is done, the postmaster sends SIGINT (previously unused) to the walsenders. This triggers the existing shutdown behavior of sending out the shutdown checkpoint record and then terminating. Author: Michael Paquier <michael.paquier@gmail.com> Reported-by: Fujii Masao <masao.fujii@gmail.com> https://git.postgresql.org/pg/commitdiff/086221cf6b1727c2baed4703c582f657b7c5350e

Andrew Dunstan pushed:

Tom Lane pushed:

  • Update time zone data files to tzdata release 2017b. DST law changes in Chile, Haiti, and Mongolia. Historical corrections for Ecuador, Kazakhstan, Liberia, and Spain. The IANA crew continue their campaign to replace invented time zone abbrevations with numeric GMT offsets. This update changes numerous zones in South America, the Pacific and Indian oceans, and some Asian and Middle Eastern zones. I kept these abbreviations in the tznames/ data files, however, so that we will still accept them for input. (We may want to start trimming those files someday, but I think we should wait for the upstream dust to settle before deciding what to do.) In passing, add MESZ (Mitteleuropaeische Sommerzeit) to the tznames lists; since we accept MEZ (Mitteleuropaeische Zeit) it seems rather strange not to take the other one. And fix some incorrect, or at least obsolete, comments that certain abbreviations are not traceable to the IANA data. https://git.postgresql.org/pg/commitdiff/74a20d0ab7c99b3efcf5dc7aac741e3b2f952a34
  • Fix mis-optimization of semijoins with more than one LHS relation. The inner-unique patch (commit 9c7f5229a) supposed that if we're considering a JOIN_UNIQUE_INNER join path, we can always set inner_unique for the join, because the inner path produced by create_unique_path should be unique relative to the outer relation. However, that's true only if we're considering joining to the whole outer relation --- otherwise we may be applying only some of the join quals, and so the inner path might be non-unique from the perspective of this join. Adjust the test to only believe that we can set inner_unique if we have the whole semijoin LHS on the outer side. There is more that can be done in this area, but this commit is only intended to provide the minimal fix needed to get correct plans. Per report from Teodor Sigaev. Thanks to David Rowley for preliminary investigation. Discussion: https://postgr.es/m/f994fc98-389f-4a46-d1bc-c42e05cb43ed@sigaev.ru https://git.postgresql.org/pg/commitdiff/2057a58d1629ebffce694e3cef7f714571a88dd7
  • Reduce semijoins with unique inner relations to plain inner joins. If the inner relation can be proven unique, that is it can have no more than one matching row for any row of the outer query, then we might as well implement the semijoin as a plain inner join, allowing substantially more freedom to the planner. This is a form of outer join strength reduction, but it can't be implemented in reduce_outer_joins() because we don't have enough info about the individual relations at that stage. Instead do it much like remove_useless_joins(): once we've built base relations, we can make another pass over the SpecialJoinInfo list and get rid of any entries representing reducible semijoins. This is essentially a followon to the inner-unique patch (commit 9c7f5229a) and makes use of the proof machinery that that patch created. We need only minor refactoring of innerrel_is_unique's API to support this usage. Per performance complaint from Teodor Sigaev. Discussion: https://postgr.es/m/f994fc98-389f-4a46-d1bc-c42e05cb43ed@sigaev.ru https://git.postgresql.org/pg/commitdiff/92a43e4857d9682b93c9f755f453cc8fd7c66c81
  • Improve function header comment for create_singleton_array(). Mentioning the caller is neither future-proof nor an adequate substitute for giving an API specification. Per gripe from Neha Khatri, though I changed the patch around some. Discussion: https://postgr.es/m/CAFO0U+_fS5SRhzq6uPG+4fbERhoA9N2+nPrtvaC9mmeWivxbsA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/54affb41e79cf4bec00bb5e00eb12a1715b9e278
  • Ensure commands in extension scripts see the results of preceding DDL. Due to a missing CommandCounterIncrement() call, parsing of a non-utility command in an extension script would not see the effects of the immediately preceding DDL command, unless that command's execution ends with CommandCounterIncrement() internally ... which some do but many don't. Report by Philippe Beaudoin, diagnosis by Julien Rouhaud. Rather remarkably, this bug has evaded detection since extensions were invented, so back-patch to all supported branches. Discussion: https://postgr.es/m/2cf7941e-4e41-7714-3de8-37b1a8f74dff@free.fr https://git.postgresql.org/pg/commitdiff/9209e07605afe0349660447f20d83ef165cdd0ae
  • Remove create_singleton_array(), hard-coding the case in its sole caller. create_singleton_array() was not really as useful as we perhaps thought when we added it. It had never accreted more than one call site, and is only saving a dozen lines of code at that one, which is considerably less bulk than the function itself. Moreover, because of its insistence on using the caller's fn_extra cache space, it's arguably a coding hazard. text_to_array_internal() does not currently use fn_extra in any other way, but if it did it would be subtly broken, since the conflicting fn_extra uses could be needed within a single query, in the seldom-tested case that the field separator varies during the query. The same objection seems likely to apply to any other potential caller. The replacement code is a bit uglier, because it hardwires knowledge of the storage parameters of type TEXT, but it's not like we haven't got dozens or hundreds of other places that do the same. Uglier seems like a good tradeoff for smaller, faster, and safer. Per discussion with Neha Khatri. Discussion: https://postgr.es/m/CAFO0U+_fS5SRhzq6uPG+4fbERhoA9N2+nPrtvaC9mmeWivxbsA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/23c6eb03360d270051bf7dcb289ecb0cd114d29f
  • Remove useless and rather expensive stanza in matview regression test. This removes a test case added by commit b69ec7cc9, which was intended to exercise a corner case involving the rule used at that time that materialized views were unpopulated iff they had physical size zero. We got rid of that rule very shortly later, in commit 1d6c72a55, but kept the test case. However, because the case now asks what VACUUM will do to a zero-sized physical file, it would be pretty surprising if the answer were ever anything but "nothing" ... and if things were indeed that broken, surely we'd find it out from other tests. Since the test involves a table that's fairly large by regression-test standards (100K rows), it's quite slow to run. Dropping it should save some buildfarm cycles, so let's do that. Discussion: https://postgr.es/m/32386.1493831320@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/4dd4104342068e542e865e5b0486e14600746221
  • Fix pfree-of-already-freed-tuple when rescanning a GiST index-only scan. GiST's getNextNearest() function attempts to pfree the previously-returned tuple if any (that is, scan->xs_hitup in HEAD, or scan->xs_itup in older branches). However, if we are rescanning a plan node after ending a previous scan early, those tuple pointers could be pointing to garbage, because they would be pointing into the scan's pageDataCxt or queueCxt which has been reset. In a debug build this reliably results in a crash, although I think it might sometimes accidentally fail to fail in production builds. To fix, clear the pointer field anyplace we reset a context it might be pointing into. This may be overkill --- I think probably only the queueCxt case is involved in this bug, so that resetting in gistrescan() would be sufficient --- but dangling pointers are generally bad news, so let's avoid them. Another plausible answer might be to just not bother with the pfree in getNextNearest(). The reconstructed tuples would go away anyway in the context resets, and I'm far from convinced that freeing them a bit earlier really saves anything meaningful. I'll stick with the original logic in this patch, but if we find more problems in the same area we should consider that approach. Per bug #14641 from Denis Smirnov. Back-patch to 9.5 where this logic was introduced. Discussion: https://postgr.es/m/20170504072034.24366.57688@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/3f074845a8c190b365e68e39e39017ff70330f2a
  • Suppress compiler warning about unportable pointer value. Setting a pointer value to "0xdeadbeef" draws a warning from some compilers, and for good reason. Be less cute and just set it to NULL. In passing make some other cosmetic adjustments nearby. Discussion: https://postgr.es/m/CAJrrPGdW3EkU-CRobvVKYf3fJuBdgWyuGeAbNzAQ4yBh+bfb_Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b3a47cdfd692079e36d2055d7d93759e083263ca
  • First-draft release notes for 9.6.3. As usual, the release notes for other branches will be made by cutting these down, but put them up for community review first. Note there are some entries that really only apply to pre-9.6 branches. https://git.postgresql.org/pg/commitdiff/54dbd4dc78b045ffcc046b9a43681770c3992dd4
  • Document current_role. This system function has been there a very long time, but somehow escaped being listed in func.sgml. Fabien Coelho and Tom Lane Discussion: https://postgr.es/m/alpine.DEB.2.20.1705061027580.3896@lancre https://git.postgresql.org/pg/commitdiff/a9c6d704354bfe91bc389742cb5d331ae4e93831
  • Second pass on 9.6.3 release notes. Improve description of logical decoding snapshot issues, per suggestion from Petr Jelinek. Mention possible need to re-sync logical replicas as a post-upgrade task. Minor copy-editing for some other items. https://git.postgresql.org/pg/commitdiff/334b82cd56a65e09154d9f930d35a761a9c5cfab
  • Restore fullname[] contents before falling through in pg_open_tzfile(). Fix oversight in commit af2c5aa88: if the shortcut open() doesn't work, we need to reset fullname[] to be just the name of the toplevel tzdata directory before we fall through into the pre-existing code. This failed to be exposed in my (tgl's) testing because the fall-through path is actually never taken under normal circumstances. David Rowley, per report from Amit Kapila Discussion: https://postgr.es/m/CAA4eK1LC7CaNhRAQ__C3ht1JVrPzaAXXhEJRnR5L6bfYHiLmWw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5788a5670e4a58049b8adc82c4fef97a2c3be327
  • Install the "posixrules" timezone link in MSVC builds. Somehow, we'd missed ever doing this. The consequences aren't too severe: basically, the timezone library would fall back on its hardwired notion of the DST transition dates to use for a POSIX-style zone name, rather than obeying US/Eastern which is the intended behavior. The net effect would only be to obey current US DST law further back than it ought to apply; so it's not real surprising that nobody noticed. David Rowley, per report from Amit Kapila Discussion: https://postgr.es/m/CAA4eK1LC7CaNhRAQ__C3ht1JVrPzaAXXhEJRnR5L6bfYHiLmWw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/d4e59c5521c244e809c3d68df51fb79543578e41
  • Guard against null t->tm_zone in strftime.c. The upstream IANA code does not guard against null TM_ZONE pointers in this function, but in our code there is such a check in the other pre-existing use of t->tm_zone. We do have some places that set pg_tm.tm_zone to NULL. I'm not entirely sure it's possible to reach strftime with such a value, but I'm not sure it isn't either, so be safe. Per Coverity complaint. https://git.postgresql.org/pg/commitdiff/a54d5875fe0bc19d05236b85e1e1bf0af9fa2902
  • Improve performance of timezone loading, especially pg_timezone_names view. tzparse() would attempt to load the "posixrules" timezone database file on each call. That might seem like it would only be an issue when selecting a POSIX-style zone name rather than a zone defined in the timezone database, but it turns out that each zone definition file contains a POSIX-style zone string and tzload() will call tzparse() to parse that. Thus, when scanning the whole timezone file tree as we do in the pg_timezone_names view, "posixrules" was read repetitively for each zone definition file. Fix that by caching the file on first use within any given process. (We cache other zone definitions for the life of the process, so there seems little reason not to cache this one as well.) This probably won't help much in processes that never run pg_timezone_names, but even one additional SET of the timezone GUC would come out ahead. An even worse problem for pg_timezone_names is that pg_open_tzfile() has an inefficient way of identifying the canonical case of a zone name: it basically re-descends the directory tree to the zone file. That's not awful for an individual "SET timezone" operation, but it's pretty horrid when we're inspecting every zone in the database. And it's pointless too because we already know the canonical spelling, having just read it from the filesystem. Fix by teaching pg_open_tzfile() to avoid the directory search if it's not asked for the canonical name, and backfilling the proper result in pg_tzenumerate_next(). In combination these changes seem to make the pg_timezone_names view about 3x faster to read, for me. Since a scan of pg_timezone_names has up to now been one of the slowest queries in the regression tests, this should help some little bit for buildfarm cycle times. Back-patch to all supported branches, not so much because it's likely that users will care much about the view's performance as because tracking changes in the upstream IANA timezone code is really painful if we don't keep all the branches in sync. Discussion: https://postgr.es/m/27962.1493671706@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/af2c5aa88d38573724e40fa029499b4db20b0eb2
  • Third pass on 9.6.3 release notes. Add updates for recent commits. In passing, credit Etsuro Fujita for his work on the postgres_fdw query cancel feature in 9.6; I seem to have missed that in the original drafting of the 9.6 notes. https://git.postgresql.org/pg/commitdiff/86713deecda4ddbf8e339ec48fafa4d8080f6079

Magnus Hagander pushed:

Álvaro Herrera pushed:

Heikki Linnakangas pushed:

Bruce Momjian pushed:

Stephen Frost pushed:

  • Change the way pg_dump retrieves partitioning info. This gets rid of the code that issued separate queries to retrieve the partitioning parent-child relationship, parent partition key, and child partition bound information. With this patch, the information is retrieved instead using the queries issued from getTables() and getInherits(), which is both more efficient than the previous approach and doesn't require any new code. Since the partitioning parent-child relationship is now retrieved with the same old code that handles inheritance, partition attributes receive a proper flagInhAttrs() treatment (that it didn't receive before), which is needed so that the inherited NOT NULL constraints are not emitted if we already emitted it for the parent. Also, fix a bug in pg_dump's --binary-upgrade code, which caused pg_dump to emit invalid command to attach a partition to its parent. Author: Amit Langote, with some additional changes by me. https://git.postgresql.org/pg/commitdiff/44c528810a1eca52a7888ed74c08353d45331b00
  • RLS: Fix ALL vs. SELECT+UPDATE policy usage. When we add the SELECT-privilege based policies to the RLS with check options (such as for an UPDATE statement, or when we have INSERT ... RETURNING), we need to be sure and use the 'USING' case if the policy is actually an 'ALL' policy (which could have both a USING clause and an independent WITH CHECK clause). This could result in policies acting differently when built using ALL (when the ALL had both USING and WITH CHECK clauses) and when building the policies independently as SELECT and UPDATE policies. Fix this by adding an explicit boolean to add_with_check_options() to indicate when the USING policy should be used, even if the policy has both USING and WITH CHECK policies on it. Reported by: Rod Taylor Back-patch to 9.5 where RLS was introduced. https://git.postgresql.org/pg/commitdiff/aa5d3c0b3fb906dfa910b0ca6f75ab701b2f1c09
  • pg_dump: Don't leak memory in buildDefaultACLCommands(). buildDefaultACLCommands() didn't destroy the string buffer created in certain cases, leading to a memory leak. Fix by destroying the buffer before returning from the function. Spotted by Coverity. Author: Michael Paquier Back-patch to 9.6 where buildDefaultACLCommands() was added. https://git.postgresql.org/pg/commitdiff/09f842181943b6e83b0779f2e872ff0180b66883

Andres Freund pushed:

Correctifs en attente

Amit Langote sent in two revisions of a patch to clarify statement trigger behavior with inheritance.

Andres Freund sent in a patch to fix initial logical decoding snapshat race condition.

Peter Eisentraut sent in a patch to pg_ctl which makes failure to complete operation a nonzero exit and remove unnecessary pg_is_in_recovery calls in tests.

Amit Langote sent in two revisions of a patch to emit "correct" range partition constraint expression.

David Rowley sent in a patch to renames all wal_location functions to wal_lsn, renames all system view columns to use "lsn" instead of "location", renames function parameters to use "lsn" instead of "location", renames function parameters "wal_position" to "lsn", changes documentation to reflect the aforementioned changes, fixes a bug where docs claimed return type of pg_logical_slot_peek_changes.location was text, when it was pg_lsn, and change some places in the func.sgml where it was referring to the lsn as a "position" rather than "location".

Amit Khandekar sent in another revision of a patch to enable UPDATEs of partition keys in declaratively partitioned tables.

Pavel Stěhule sent in a patch to add a \gdesc option to psql.

Peter Eisentraut sent in a patch to rename transaction log to write-ahead log.

Haribabu Kommi sent in a patch to ensure that pg_basebackup treats STDOUT correctly on Windows.

Rahila Syed sent in two more revisions of a patch to add support for default partitions in declaratively partitioned tables.

Petr Jelínek sent in a patch to fix statistics reporting in logical replication workers.

Thomas Munro sent in a patch to allow transition tables in AFTER triggers in one relation only.

Amul Sul sent in another revision of a patch to implement declarative hash partitioning.

Marina Polyakova sent in two revisions of a patch to create the infrastructure for precalculating stable functions.

Álvaro Herrera sent in two more revisions of a patch to add a WITH clause to CREATE STATISTICS.

Stas Kelvich sent in another revision of a patch to trim some logical replication ApplyContext bloat.

Robert Haas sent in two revisions of a patch to improve PostgreSQL FDW abort behavior.

Magnus Hagander sent in a patch to add pg_move_replication_slot().

Nikita Glukhov sent in a patch to fix freeing of dangling IndexScanDesc.xs_hitup in GiST.

Andres Freund sent in another revision of a patch to fix an off-by-one around GetLastImportantRecPtr.

David Rowley sent in a patch to implement parallel nextpage batching.

Thomas Munro sent in a patch to fix named tuplestore rescan.

Heikki Linnakangas sent in a patch to remove support for password_encryption='off' / 'plain'.

Dmitriy Sarafannikov sent in a patch to implement a new type of snapshot that accepts any non-vacuumable tuples.

Petr Jelínek sent in a patch to rework the options for logical replication.

Alexander Korotkov sent in another revision of a patch to implement incremental sort.

Petr Jelínek sent in a patch to check connection info in ALTER SUBSCRIPTION.

David Rowley sent in a doc patch to caution about CTE changes in the future, when the now-mandatory optimization fence may go away.

Stephen Frost sent in a patch to fix ALL RLS policy using check.

Petr Jelínek sent in a patch to remove the NODROP SLOT option from DROP SUBSCRIPTION.

David Rowley sent in a patch to use atomics for heap_parallelscan_nextpage().

Bruce Momjian sent in another revision of a patch to update the release notes for PostgreSQL 10.

par chl le jeudi 18 mai 2017 à 22h40

Loxodata

PostgreSQL 10 en bêta

Publication de PostgreSQL 10 Bêta 1

Le PostgreSQL Global Development Group annonce aujourd’hui la première bêta de la version 10 de PostgreSQL. Cette version contient une pré-version de toutes les fonctionnalités disponibles dans la version finale. Quelques modifications peuvent encore intervenir.

Les utilisateurs peuvent désormais tester leurs applications avec cette version en prévision de la version finale.

Principales fonctionnalités de la version 10

La nouvelle version contient de nombreuses fonctionnalités nouvelles. Ces fonctionnalités faciliteront les extensions internes (scale up) et externes (scale out) des infrastructures PostgreSQL :

  • Réplication logique : Option intégrée pour la réplication de tables spécifiques ou la migration ;
  • Partitionnement de table natif : partitionnement par liste ou intervalle comme objets natifs ;
  • Parallélisme de requête additionnel : inclusion des parcours d’index, bitmap et merge joins ;
  • Quorum Commit pour la réplication synchrone : s’assurer contre la perte de plusieurs nœuds.

Cette version inclut également 3 améliorations concernant les connexions à PostgreSQL, que nous demandons aux auteurs de connecteurs de supporter, et aux utilisateurs de tester :

  • Authentification SCRAM, pour des accès par mot de passe plus sûrs ;
  • “Failover” multi-hôtes, connexion au premier nœud disponible dans une liste d’hôtes ;
  • Paramètre target_session_attrs, pour permettre au client de demander un hôte en lecture/écriture.

Fonctionnalités complémentaires

De nombreuses autres fonctionnalités et améliorations ont été ajoutées à PostgreSQL 10. En fonction des utilisateurs, elles peuvent paraître plus ou moins importantes que les précédentes. Mais elles doivent toutes être testées.

Citons :

  • Index Hash résistants aux crash, et réplicables ;
  • Statistiques multi-colonnes corrélées ;
  • Nouveaux rôles de “monitoring” pour les octrois de permissions ;
  • Temps d’attente de verrou dans pg_stat_activity ;
  • expression de requête XMLTABLE ;
  • Politiques restrictives pour Row Level Security ;
  • Support Full Text Search pour JSON et JSONB;
  • Support de la compression support pour pg_receivewal ;
  • Support des collations ICU ;
  • Push Down Aggregates pour les serveurs externes ;
  • Transition Tables dans l’exécution des triggers.

De plus, les développeurs ont contribué à l’amélioration des performances de la fonction SUM(), de la conversion d’encodage de caractères, de l’évaluation d’expression, des grouping sets, et des jointures sur des colonnes uniques.

Les requêtes analytiques sur des forts volumes de données devraient être 40% plus rapides. Testez et vérifiez par vous-même, et dites-nous ce qu’il en est.

la liste complète des fonctionnalités nouvelles ou modifiées est dans les notes de révision : https://www.postgresql.org/docs/devel/static/release-10.html

Tests de compatibilité et recherche de bogues

Nous comptons sur vous pour tester cette version dans vos cas d’usage. Cela permettra de découvrir d’éventuels bogues ou régression avant la sortie de la PostgreSQL 10. S’agissant d’une version bêta, il est possible qu’interviennent des modifications dans le comportement des bases, les détails fonctionnels, et les API.

Vos tests et vos retours nous aideront à finaliser ces nouvelles fonctionnalités. De la qualité des tests utilisateurs dépend la date de publication de la version finale.

Enfin, la version 10 contient de nombreuses modifications incompatibles avec les versions majeures qui précédent. En particulier le renommage de “xlog” en “wal” et la modification de la numérotation des versions.

Nous encourageons les utilisateurs à tester leurs applications, scripts, et plateformes rapidement.

Les notes de version et page de nouveautés présentent les détails de cette version.

Agenda

Il s’agit de la première version bêta pour la version 10. Le projet PostgreSQL publiera d’autres bêta, si nécessaire, puis une ou plusieurs release candidates, jusqu’à la version finale, fin 2017. Pour plus d’informations et des suggestions sur la manière de tester les bêta, on peut se référer à la page Beta Testing : https://www.postgresql.org/developer/beta.

La documentation complête et les notes de version sont disponibles en ligne, et installées avec PostgreSQL.

Liens

par contact@loxodata.com (Loxodata) le jeudi 18 mai 2017 à 15h26

samedi 6 mai 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 30 avril 2017

Ubuntu Zesty (17.04) et Debian stretch (9) sont maintenant prises en charge par apt.postgresql.org : https://wiki.postgresql.org/wiki/Apt

[ndt: MeetUp à Lyon le 28 juin :https://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170430221936.GA9169@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Code review for commands/statscmds.c. Fix machine-dependent sorting of column numbers. (Odd behavior would only materialize for column numbers above 255, but that's certainly legal.) Fix poor choice of SQLSTATE for some errors, and improve error message wording. (Notably, "is not a scalar type" is a totally misleading way to explain "does not have a default btree opclass".) Avoid taking AccessExclusiveLock on the associated relation during DROP STATISTICS. That's neither necessary nor desirable, and it could easily have put us into situations where DROP fails (compare commit 68ea2b7f9). Adjust/improve comments. David Rowley and Tom Lane Discussion: https://postgr.es/m/CAKJS1f-GmCfPvBbAEaM5xoVOaYdVgVN1gicALSoYQ77z-+vLbw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4b34624daadd9837cd65f20419f832b295c67ecb
  • Fix postmaster's handling of fork failure for a bgworker process. This corner case didn't behave nicely at all: the postmaster would (partially) update its state as though the process had started successfully, and be quite confused thereafter. Fix it to act like the worker had crashed, instead. In passing, refactor so that do_start_bgworker contains all the state-change logic for bgworker launch, rather than just some of it. Back-patch as far as 9.4. 9.3 contains similar logic, but it's just enough different that I don't feel comfortable applying the patch without more study; and the use of bgworkers in 9.3 was so small that it doesn't seem worth the extra work. transam/parallel.c is still entirely unprepared for the possibility of bgworker startup failure, but that seems like material for a separate patch. Discussion: https://postgr.es/m/4905.1492813727@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/4fe04244b584749351657e99f3e6e1436e9b65a8
  • Run the postmaster's signal handlers without SA_RESTART. The postmaster keeps signals blocked everywhere except while waiting for something to happen in ServerLoop(). The code expects that the select(2) will be cancelled with EINTR if an interrupt occurs; without that, followup actions that should be performed by ServerLoop() itself will be delayed. However, some platforms interpret the SA_RESTART signal flag as meaning that they should restart rather than cancel the select(2). Worse yet, some of them restart it with the original timeout delay, meaning that a steady stream of signal interrupts can prevent ServerLoop() from iterating at all if there are no incoming connection requests. Observable symptoms of this, on an affected platform such as HPUX 10, include extremely slow parallel query startup (possibly as much as 30 seconds) and failure to update timestamps on the postmaster's sockets and lockfiles when no new connections arrive for a long time. We can fix this by running the postmaster's signal handlers without SA_RESTART. That would be quite a scary change if the range of code where signals are accepted weren't so tiny, but as it is, it seems safe enough. (Note that postmaster children do, and must, reset all the handlers before unblocking signals; so this change should not affect any child process.) There is talk of rewriting the postmaster to use a WaitEventSet and not do signal response work in signal handlers, at which point it might be appropriate to revert this patch. But that's not happening before v11 at the earliest. Back-patch to 9.6. The problem exists much further back, but the worst symptom arises only in connection with parallel query, so it does not seem worth taking any portability risks in older branches. Discussion: https://postgr.es/m/9205.1492833041@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/8939020853e63da6b4f5a63f023b02776a441b5d
  • Use pselect(2) not select(2), if available, to wait in postmaster's loop. Traditionally we've unblocked signals, called select(2), and then blocked signals again. The code expects that the select() will be cancelled with EINTR if an interrupt occurs; but there's a race condition, which is that an already-pending signal will be delivered as soon as we unblock, and then when we reach select() there will be nothing preventing it from waiting. This can result in a long delay before we perform any action that ServerLoop was supposed to have taken in response to the signal. As with the somewhat-similar symptoms fixed by commit 893902085, the main practical problem is slow launching of parallel workers. The window for trouble is usually pretty short, corresponding to one iteration of ServerLoop; but it's not negligible. To fix, use pselect(2) in place of select(2) where available, as that's designed to solve exactly this problem. Where not available, we continue to use the old way, and are no worse off than before. pselect(2) has been required by POSIX since about 2001, so most modern platforms should have it. A bigger portability issue is that some implementations are said to be non-atomic, ie pselect() isn't really any different from unblock/select/reblock. Still, we're no worse off than before on such a platform. There is talk of rewriting the postmaster to use a WaitEventSet and not do signal response work in signal handlers, at which point this could be reverted, since we'd be using a self-pipe to solve the race condition. But that's not happening before v11 at the earliest. Back-patch to 9.6. The problem exists much further back, but the worst symptom arises only in connection with parallel query, so it does not seem worth taking any portability risks in older branches. Discussion: https://postgr.es/m/9205.1492833041@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/81069a9efc5a374dd39874a161f456f0fb3afba4
  • Revert "Use pselect(2) not select(2), if available, to wait in postmaster's loop.". This reverts commit 81069a9efc5a374dd39874a161f456f0fb3afba4. Buildfarm results suggest that some platforms have versions of pselect(2) that are not merely non-atomic, but flat out non-functional. Revert the use-pselect patch to confirm this diagnosis (and exclude the no-SA_RESTART patch as the source of trouble). If it's so, we should probably look into blacklisting specific platforms that have broken pselect. Discussion: https://postgr.es/m/9696.1493072081@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/64925603c972aa3a9f1d4c42686dba67f1a7e9d0
  • Silence compiler warning induced by commit de4389712. Smarter compilers can see that "slot" can't be used uninitialized, but some popular ones cannot. Noted by Jeff Janes. https://git.postgresql.org/pg/commitdiff/49da00677dc25d83be44372918e21a405863ace2
  • Allow multiple bgworkers to be launched per postmaster iteration. Previously, maybe_start_bgworker() would launch at most one bgworker process per call, on the grounds that the postmaster might otherwise neglect its other duties for too long. However, that seems overly conservative, especially since bad effects only become obvious when many hundreds of bgworkers need to be launched at once. On the other side of the coin is that the existing logic could result in substantial delay of bgworker launches, because ServerLoop isn't guaranteed to iterate immediately after a signal arrives. (My attempt to fix that by using pselect(2) encountered too many portability question marks, and in any case could not help on platforms without pselect().) One could also question the wisdom of using an O(N^2) processing method if the system is intended to support so many bgworkers. As a compromise, allow that function to launch up to 100 bgworkers per call (and in consequence, rename it to maybe_start_bgworkers). This will allow any normal parallel-query request for workers to be satisfied immediately during sigusr1_handler, avoiding the question of whether ServerLoop will be able to launch more promptly. There is talk of rewriting the postmaster to use a WaitEventSet to avoid the signal-response-delay problem, but I'd argue that this change should be kept even after that happens (if it ever does). Backpatch to 9.6 where parallel query was added. The issue exists before that, but previous uses of bgworkers typically aren't as sensitive to how quickly they get launched. Discussion: https://postgr.es/m/4707.1493221358@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/aa1351f1eec4adae39be59ce9a21410f9dd42118
  • Make latch.c more paranoid about child-process cases. Although the postmaster doesn't currently create a self-pipe or any latches, there's discussion of it doing so in future. It's also conceivable that a shared_preload_libraries extension would try to create such a thing in the postmaster process today. In that case the self-pipe FDs would be inherited by forked child processes. latch.c was entirely unprepared for such a case and could suffer an assertion failure, or worse try to use the inherited pipe if somebody called WaitLatch without having called InitializeLatchSupport in that process. Make it keep track of whether InitializeLatchSupport has been called in the *current* process, and do the right thing if state has been inherited from a parent. Apply FD_CLOEXEC to file descriptors created in latch.c (the self-pipe, as well as epoll event sets). This ensures that child processes spawned in backends, the archiver, etc cannot accidentally or intentionally mess with these FDs. It also ensures that we end up with the right state for the self-pipe in EXEC_BACKEND processes, which otherwise wouldn't know to close the postmaster's self-pipe FDs. Back-patch to 9.6, mainly to keep latch.c looking similar in all branches it exists in. Discussion: https://postgr.es/m/8322.1493240739@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/fa31b6f4e9696f3c9777bf4ec2faea822826ce9f
  • Cope with glibc too old to have epoll_create1(). Commit fa31b6f4e supposed that we didn't have to worry about that anymore, but it seems that RHEL5 is like that, and that's still a supported platform. Put back the prior coding under an #ifdef, adding an explicit fcntl() to retain the desired CLOEXEC property. Discussion: https://postgr.es/m/12307.1493325329@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/82ebbeb0abfe40fe5f19a6fcdffc7484fd3a35b0
  • Avoid slow shutdown of pg_basebackup. pg_basebackup's child process did not pay any attention to the pipe from its parent while waiting for input from the source server. If no server data was arriving, it would only wake up and check the pipe every standby_message_timeout or so. This creates a problem since the parent process might determine and send the desired stop position only after the server has reached end-of-WAL and stopped sending data. In the src/test/recovery regression tests, the timing is repeatably such that it takes nearly 10 seconds for the child process to realize that it should shut down. It's not clear how often that would happen in real-world cases, but it sure seems like a bug --- and if the user turns off standby_message_timeout or sets it very large, the delay could be a lot worse. To fix, expand the StreamCtl API to allow the pipe input FD to be passed down to the low-level wait routine, and watch both sockets when sleeping. (Note: AFAICS this issue doesn't affect the Windows port, since it doesn't rely on a pipe to transfer the stop position to the child thread.) Discussion: https://postgr.es/m/6456.1493263884@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/7834d20b57a4320308c3f8262fabf898f89e6a71
  • Micro-optimize some slower queries in the opr_sanity regression test. Convert the binary_coercible() and physically_coercible() functions from SQL to plpgsql. It's not that plpgsql is inherently better at doing queries; if you simply convert the previous single SQL query into one RETURN expression, it's no faster. The problem with the existing code is that it fools the plancache into deciding that it's worth re-planning the query every time, since constant-folding with a concrete value for $2 allows elimination of at least one sub-SELECT. In reality that's using the planner to do the equivalent of a few runtime boolean tests, causing the function to run much slower than it should. Splitting the AND/OR logic into separate plpgsql statements allows each if-expression to acquire a static plan. Also, get rid of some uses of obj_description() in favor of explicitly joining to pg_description, allowing the joins to be optimized better. (Someday we might improve the SQL-function-inlining logic enough that this happens automatically, but today is not that day.) Together, these changes reduce the runtime of the opr_sanity regression test by about a factor of two on one of my slower machines. They don't seem to help as much on a fast machine, but this should at least benefit the buildfarm. https://git.postgresql.org/pg/commitdiff/c23844212d768b0423859437ca8189b89fd85250
  • Fix possible null pointer dereference or invalid warning message. Thinko in commit de4389712: this warning message references the wrong "LogicalRepWorker *" variable. This would often result in a core dump, but if it didn't, the message would show the wrong subscription OID. In passing, adjust the message text to format a subscription OID similarly to how that's done elsewhere in the function; and fix grammatical issues in some nearby messages. Per Coverity testing. https://git.postgresql.org/pg/commitdiff/12d11432b4db8a2ae665287e05f0f6868d35545e
  • Sync our copy of the timezone library with IANA release tzcode2017b. zic no longer mishandles some transitions in January 2038 when it attempts to work around Qt bug 53071. This fixes a bug affecting Pacific/Tongatapu that was introduced in zic 2016e. localtime.c now contains a workaround, useful when loading a file generated by a buggy zic. There are assorted cosmetic changes as well, notably relocation of a bunch of #defines. https://git.postgresql.org/pg/commitdiff/e18b2c480da478f62781e06488cda56fe1b4e919

Fujii Masao pushed:

Bruce Momjian pushed:

Peter Eisentraut pushed:

  • postgres_fdw: Fix join push down with extensions. Objects in an extension are shippable to a foreign server if the extension is part of the foreign server definition's shippable extensions list. But this was not properly considered in some cases when checking whether a join condition can be pushed to a foreign server and the join condition uses an object from a shippable extension. So the join would never be pushed down in those cases. So, the list of extensions needs to be made available in fpinfo of the relation being considered to be pushed down before any expressions are assessed for being shippable. Fix foreign_join_ok() to do that for a join relation. The code to save FDW options in fpinfo is scattered at multiple places. Bring all of that together into functions apply_server_options(), apply_table_options(), and merge_fdw_options(). David Rowley and Ashutosh Bapat, per report from David Rowley https://git.postgresql.org/pg/commitdiff/332bec1e6096679b04ec9717beb44a858495260f
  • Wake up launcher when enabling a subscription. Otherwise one would have to wait up to DEFAULT_NAPTIME_PER_CYCLE until the subscription worker is considered for starting. There is a small race condition: If one enables a subscription right after disabling it, the launcher might not have registered the stopping when receiving the wakeup signal for the re-enabling. The start will then not happen right away but after the full cycle time. Author: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> https://git.postgresql.org/pg/commitdiff/a3f17b9c315512e9c116dbc243056aad8b372e18
  • Fix various concurrency issues in logical replication worker launching. The code was originally written with assumption that launcher is the only process starting the worker. However that hasn't been true since commit 7c4f52409 which failed to modify the worker management code adequately. This patch adds an in_use field to the LogicalRepWorker struct to indicate whether the worker slot is being used and uses proper locking everywhere this flag is set or read. However if the parent process dies while the new worker is starting and the new worker fails to attach to shared memory, this flag would never get cleared. We solve this rare corner case by adding a sort of garbage collector for in_use slots. This uses another field in the LogicalRepWorker struct named launch_time that contains the time when the worker was started. If any request to start a new worker does not find free slot, we'll check for workers that were supposed to start but took too long to actually do so, and reuse their slot. In passing also fix possible race conditions when stopping a worker that hasn't finished starting yet. Author: Petr Jelinek <petr.jelinek@2ndquadrant.com> Reported-by: Fujii Masao <masao.fujii@gmail.com> https://git.postgresql.org/pg/commitdiff/de4389712206d2686e09ad8d6dd112dc4b6c6d42
  • doc: ALTER SUBSCRIPTION documentation fixes. WITH is optional for REFRESH PUBLICATION. Also, remove a spurious bracket and fix a punctuation. Author: Euler Taveira <euler@timbira.com.br> https://git.postgresql.org/pg/commitdiff/e315346d839ef27f9d5f2584f44de09f08573df2
  • Fix query that gets remote relation info. Publisher relation can be incorrectly chosen, if there are more than one relation in different schemas with the same name. Author: Euler Taveira <euler@timbira.com.br> https://git.postgresql.org/pg/commitdiff/61ecc90be624e699164a8d3efb291e267b711142
  • Spelling fixes in code comments. Author: Euler Taveira <euler@timbira.com.br> https://git.postgresql.org/pg/commitdiff/e495c1683f2c243f6769b34a009cf9c28526b555
  • Fix typo in comment. Author: Masahiko Sawada <sawada.mshk@gmail.com> https://git.postgresql.org/pg/commitdiff/6c9bd27aece5efd0ccef668828a44f59bd2c7a44
  • Wait between tablesync worker restarts. Before restarting a tablesync worker for the same relation, wait wal_retrieve_retry_interval (currently 5s by default). This avoids restarting failing workers in a tight loop. We keep the last start times in a hash table last_start_times that is separate from the table_states list, because that list is cleared out on syscache invalidation, which happens whenever a table finishes syncing. The hash table is kept until all tables have finished syncing. A future project might be to unify these two and keep everything in one data structure, but for now this is a less invasive change to accomplish the original purpose. For the test suite, set wal_retrieve_retry_interval to its minimum value, to not increase the test suite run time. Reviewed-by: Petr Jelinek <petr.jelinek@2ndquadrant.com> Reported-by: Masahiko Sawada <sawada.mshk@gmail.com> https://git.postgresql.org/pg/commitdiff/e3cf708016ca6045dc1cd5a0768cfecf17caf3d1
  • psql: Support identity columns in sequence display. Where the footer for an owned serial sequence would say "Owned by", put something analogous for a sequence belonging to an identity column. Reported-by: Vitaly Burovoy <vitaly.burovoy@gmail.com> https://git.postgresql.org/pg/commitdiff/e4fddfd49241dc8dfda354993bad8d5518df1873
  • doc: Fix typo in 9.6 release notes. Author: Huong Dangminh <huo-dangminh@ys.jp.nec.com> https://git.postgresql.org/pg/commitdiff/bc920bee296ec4c1e8cd1598c71f21d80a59d351

Robert Haas pushed:

Stephen Frost pushed:

  • Allow ALTER TABLE ONLY on partitioned tables. There is no need to forbid ALTER TABLE ONLY on partitioned tables, when no partitions exist yet. This can be handy for users who are building up their partitioned table independently and will create actual partitions later. In addition, this is how pg_dump likes to operate in certain instances. Author: Amit Langote, with some error message word-smithing by me https://git.postgresql.org/pg/commitdiff/9139aa19423b736470f669e566f8ef6a7f19b801
  • pg_get_partkeydef: return NULL for non-partitions. Our general rule for pg_get_X(oid) functions is to simply return NULL when passed an invalid or inappropriate OID. Teach pg_get_partkeydef to do this also, making it easier for users to use this function when querying against tables with both partitions and non-partitions (such as pg_class). As a concrete example, this makes pg_dump's life a little easier. Author: Amit Langote https://git.postgresql.org/pg/commitdiff/0c76c2463e8ab4cfd633ad8de259050e3f28b78f
  • Remove unnecessairly duplicated gram.y productions. Declarative partitioning duplicated the TypedTableElement productions, evidently to remove the need to specify WITH OPTIONS when creating partitions. Instead, simply make WITH OPTIONS optional in the TypedTableElement production and remove all of the duplicate PartitionElement-related productions. This change simplifies the syntax and makes WITH OPTIONS optional when adding defaults, constraints or storage parameters to columns when creating either typed tables or partitions. Also update pg_dump to no longer include WITH OPTIONS, since it's not necessary, and update the documentation to reflect that WITH OPTIONS is now optional. https://git.postgresql.org/pg/commitdiff/b9a3ef55b253d885081c2d0e9dc45802cab71c7b

Simon Riggs pushed:

Andres Freund pushed:

  • Preserve required !catalog tuples while computing initial decoding snapshot. The logical decoding machinery already preserved all the required catalog tuples, which is sufficient in the course of normal logical decoding, but did not guarantee that non-catalog tuples were preserved during computation of the initial snapshot when creating a slot over the replication protocol. This could cause a corrupted initial snapshot being exported. The time window for issues is usually not terribly large, but on a busy server it's perfectly possible to it hit it. Ongoing decoding is not affected by this bug. To avoid increased overhead for the SQL API, only retain additional tuples when a logical slot is being created over the replication protocol. To do so this commit changes the signature of CreateInitDecodingContext(), but it seems unlikely that it's being used in an extension, so that's probably ok. In a drive-by fix, fix handling of ReplicationSlotsComputeRequiredXmin's already_locked argument, which should only apply to ProcArrayLock, not ReplicationSlotControlLock. Reported-By: Erik Rijkers Analyzed-By: Petr Jelinek Author: Petr Jelinek, heavily editorialized by Andres Freund Reviewed-By: Andres Freund Discussion: https://postgr.es/m/9a897b86-46e1-9915-ee4c-da02e4ff6a95@2ndquadrant.com Backport: 9.4, where logical decoding was introduced. https://git.postgresql.org/pg/commitdiff/2bef06d51646058c6bb480fcdbffb1f0cc914fed
  • Don't use on-disk snapshots for exported logical decoding snapshot. Logical decoding stores historical snapshots on disk, so that logical decoding can restart without having to reconstruct a snapshot from scratch (for which the resources are not guaranteed to be present anymore). These serialized snapshots were also used when creating a new slot via the walsender interface, which can export a "full" snapshot (i.e. one that can read all tables, not just catalog ones). The problem is that the serialized snapshots are only useful for catalogs and not for normal user tables. Thus the use of such a serialized snapshot could result in an inconsistent snapshot being exported, which could lead to queries returning wrong data. This would only happen if logical slots are created while another logical slot already exists. Author: Petr Jelinek Reviewed-By: Andres Freund Discussion: https://postgr.es/m/f37e975c-908f-858e-707f-058d3b1eb214@2ndquadrant.com Backport: 9.4, where logical decoding was introduced. https://git.postgresql.org/pg/commitdiff/56e19d938dd1457ae078304df1b9903509a0a2bf
  • Don't build full initial logical decoding snapshot if NOEXPORT_SNAPSHOT. Earlier commits (56e19d938dd14 and 2bef06d5164) make it cheaper to create a logical slot if not exporting the initial snapshot. If NOEXPORT_SNAPSHOT is specified, we can skip the overhead, not just when creating a slot via sql (which can't export snapshots). As NOEXPORT_SNAPSHOT has only recently been introduced, this shouldn't be backpatched. https://git.postgresql.org/pg/commitdiff/ab9c43381ef7a7333086107847413e0b593854d0

Heikki Linnakangas pushed:

  • Misc SCRAM code cleanups. * Move computation of SaltedPassword to a separate function from scram_ClientOrServerKey(). This saves a lot of cycles in libpq, by computing SaltedPassword only once per authentication. (Computing SaltedPassword is expensive by design.) * Split scram_ClientOrServerKey() into two functions. Improves readability, by making the calling code less verbose. * Rename "server proof" to "server signature", to better match the nomenclature used in RFC 5802. * Rename SCRAM_SALT_LEN to SCRAM_DEFAULT_SALT_LEN, to make it more clear that the salt can be of any length, and the constant only specifies how long a salt we use when we generate a new verifier. Also rename SCRAM_ITERATIONS_DEFAULT to SCRAM_DEFAULT_ITERATIONS, for consistency. These things caught my eye while working on other upcoming changes. https://git.postgresql.org/pg/commitdiff/d981074c24d2f1e4f44bc6d80e967e523ce64f50

Correctifs en attente

Andrew Dunstan sent in a patch to fix TAP tests on the MSYS build system.

Jan Mich�lek sent in another revision of a patch to add mediawiki and rst formats for tables in psql.

Haribabu Kommi sent in a patch to add build support for Visual Studio 2017.

Andrew Dunstan sent in two revisions of a patch to add vcregress support for single TAP test.

Micha�l Paquier sent in two revisions of a patch to fix an issue with logical replication and PANIC during shutdown checkpoint in publisher.

Surafel Temesgen sent in a patch to add DELETE and UPDATE with LIMIT and ORDER BY.

Daniel Gustafsson sent in two more revisions of a patch to help debug TAP test failures.

Andres Freund sent in a POC patch to unify the SQL and replication grammars.

Fabien COELHO sent in two more revisions of a patch to fix pgbench TAP tests.

Rahila Syed sent in another revision of a patch to enable creating a default partition for declaratively partitioned tables.

Ashutosh Bapat sent in another revision of a patch to speed up partition-wise JOIN between declaratively partitioned tables.

Thomas Munro sent in a patch to allow creating transition tables for triggers on foreign tables and views.

Thomas Munro sent in another revision of a patch to implement [[parallel] shared] hash.

Amit Langote sent in a patch to fix VALIDATE CONSTRAINT to consider NO INHERIT attribute.

Tom Lane sent in a patch to fix the inefficient shutdown of pg_basebackup.

Alexander Korotkov sent in another revision of a patch to implement incremental sort.

Antonin Houska sent in two revisions of a patch to enable partition-wise aggregation/grouping.

Konstantin Knizhnik sent in two more revisions of a patch to auto-prepare queries similar to previous queries.

Amit Langote sent in two revisions of a patch to fire per-statement triggers of partitioned tables.

Dagfinn Ilmari Manns�ker sent in a patch to fix some release note links.

Kyotaro HORIGUCHI sent in a patch to fix a bug where PQhost may return socket dir for network connection.

Antonin Houska sent in another revision of a patch to do aggregate push-down.

Tom Lane sent in two revisions of a patch to fix a bug in conversion from EXISTS to an inner join.

Kyotaro HORIGUCHI sent in a patch to fix a behavior change in .pgpass.

Petr Jel�nek sent in another revision of a patch to add support for time based lag tracking to logical replication.

par chl le samedi 6 mai 2017 à 19h06

lundi 17 avril 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 16 avril 2017

Le PGDay 2017 argentin sera tenu à Santa Fe le 9 juin 2017. L'appel à conférenciers est lancé : http://www.pgday.com.ar

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170416230354.GA24896@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Joe Conway pushed:

Heikki Linnakangas pushed:

  • Document the "replication" option in StartupMessage. It is documented in the Streaming Replication Protocol section, but was missing from the list of options in StartupMessage description. http://git.postgresql.org/pg/commitdiff/6c4ad8b7bf420a6f598e4b45560cffc40ded0875
  • Fix indentation. Oops, I forgot to "git add" this to previous commit. http://git.postgresql.org/pg/commitdiff/9cf5c31964315181e475fc37a5e9ad2204fe3484
  • Minor cleanup of backend SCRAM code. Free each SASL message after sending it. It's not a lot of wasted memory, and it's short-lived, but the authentication code in general tries to pfree() stuff, so let's follow the example. Adding the pfree() revealed a little bug in build_server_first_message(). It attempts to keeps a copy of the sent message, but it was missing a pstrdup(), so the pointer started to dangle, after adding the pfree() into CheckSCRAMAuth(). Reword comments and debug messages slightly, while we're at it. Reviewed by Michael Paquier. Discussion: https://www.postgresql.org/message-id/6490b975-5ee1-6280-ac1d-af975b19fb9a@iki.fi http://git.postgresql.org/pg/commitdiff/00707fa58275e370dc445fa7e1130085aa04f37b
  • Improve the SASL authentication protocol. This contains some protocol changes to SASL authentiation (which is new in v10): * For future-proofing, in the AuthenticationSASL message that begins SASL authentication, provide a list of SASL mechanisms that the server supports, for the client to choose from. Currently, it's always just SCRAM-SHA-256. * Add a separate authentication message type for the final server->client SASL message, which the client doesn't need to respond to. This makes it unambiguous whether the client is supposed to send a response or not. The SASL mechanism should know that anyway, but better to be explicit. Also, in the server, support clients that don't send an Initial Client response in the first SASLInitialResponse message. The server is supposed to first send an empty request in that case, to which the client will respond with the data that usually comes in the Initial Client Response. libpq uses the Initial Client Response field and doesn't need this, and I would assume any other sensible implementation to use Initial Client Response, too, but let's follow the SASL spec. Improve the documentation on SASL authentication in protocol. Add a section describing the SASL message flow, and some details on our SCRAM-SHA-256 implementation. Document the different kinds of PasswordMessages that the frontend sends in different phases of SASL authentication, as well as GSS/SSPI authentication as separate message formats. Even though they're all 'p' messages, and the exact format depends on the context, describing them as separate message formats makes the documentation more clear. Reviewed by Michael Paquier and Álvaro Hernández Tortosa. Discussion: https://www.postgresql.org/message-id/CAB7nPqS-aFg0iM3AQOJwKDv_0WkAedRjs1W2X8EixSz+sKBXCQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4f3b87ab780b95c2cc8a591259baefaff4852037
  • Refactor libpq authentication request processing. Move the responsibility of reading the data from the authentication request message from PQconnectPoll() to pg_fe_sendauth(). This way, PQconnectPoll() doesn't need to know about all the different authentication request types, and we don't need the extra fields in the pg_conn struct to pass the data from PQconnectPoll() to pg_fe_sendauth() anymore. Reviewed by Michael Paquier. Discussion: https://www.postgresql.org/message-id/6490b975-5ee1-6280-ac1d-af975b19fb9a%40iki.fi http://git.postgresql.org/pg/commitdiff/61bf96cab06390fec66405d3caad789f4417f25a

Tom Lane pushed:

  • Move isolationtester's is-blocked query into C code for speed. Commit 4deb41381 modified isolationtester's query to see whether a session is blocked to also check for waits occurring in GetSafeSnapshot. However, it did that in a way that enormously increased the query's runtime under CLOBBER_CACHE_ALWAYS, causing the buildfarm members that use that to run about four times slower than before, and in some cases fail entirely. To fix, push the entire logic into a dedicated backend function. This should actually reduce the CLOBBER_CACHE_ALWAYS runtime from what it was previously, though I've not checked that. In passing, expose a SQL function to check for safe-snapshot blockage, comparable to pg_blocking_pids. This is more or less free given the infrastructure built to solve the other problem, so we might as well. Thomas Munro Discussion: https://postgr.es/m/20170407165749.pstcakbc637opkax@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/511540dadf1166d80b864f63979178f324844060
  • Improve castNode notation by introducing list-extraction-specific variants. This extends the castNode() notation introduced by commit 5bcab1114 to provide, in one step, extraction of a list cell's pointer and coercion to a concrete node type. For example, "lfirst_node(Foo, lc)" is the same as "castNode(Foo, lfirst(lc))". Almost half of the uses of castNode that have appeared so far include a list extraction call, so this is pretty widely useful, and it saves a few more keystrokes compared to the old way. As with the previous patch, back-patch the addition of these macros to pg_list.h, so that the notation will be available when back-patching. Patch by me, after an idea of Andrew Gierth's. Discussion: https://postgr.es/m/14197.1491841216@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/8f0530f58061b185dc385df42e62d78a18d4ae3e
  • Fix pgbench's --progress-timestamp option to print Unix-epoch timestamps. As a consequence of commit 1d63f7d2d, on platforms with CLOCK_MONOTONIC, you got some random timescale or other instead of standard Unix timestamps as expected. I'd attempted to fix pgbench for that change in commits 74baa1e3b and 67a875355, but missed this place. Fix in the same way as those previous commits, ie, just eat the cost of an extra gettimeofday(); one extra syscall per progress report isn't worth sweating over. Per report from Jeff Janes. In passing, use snprintf not sprintf for this purpose. I don't think there's any chance of actual buffer overrun, but it just looks safer. Discussion: https://postgr.es/m/CAMkU=1zrQaPwBN+NcBd3pWCb=vWaiL=mmWfJjDJjh-a7eVr-Og@mail.gmail.com http://git.postgresql.org/pg/commitdiff/feffa0e0795a5a99324890a6dd548ba162ec104c
  • Handle restriction clause lists more uniformly in postgres_fdw. Clauses in the lists retained by postgres_fdw during planning were sometimes bare boolean clauses, sometimes RestrictInfos, and sometimes a mixture of the two in the same list. The comment about that situation didn't come close to telling the full truth, either. Aside from being confusing, this had a couple of bad practical consequences: * waste of planning cycles due to inability to cache per-clause selectivity and cost estimates; * sometimes, RestrictInfos would sneak into the fdw_private list of a finished Plan node, causing failures if, for example, we tried to ship the Plan tree to a parallel worker. (It may well be that it's a bug in the parallel-query logic that we would ever try to ship such a plan to a parallel worker, but in any case this deserves to be cleaned up.) To fix, rearrange so that clause lists in PgFdwRelationInfo are always lists of RestrictInfos, and then strip the RestrictInfos at the last minute when making a Plan node. In passing do a bit of refactoring and comment cleanup in postgresGetForeignPlan and foreign_join_ok. Although the messiness here dates back at least to 9.6, there's no evidence that it causes anything worse than wasted planning cycles in 9.6, so no back-patch for now. Per fuzz testing by Andreas Seltenreich. Tom Lane and Ashutosh Bapat Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/28b047875554b29837cded70a19384dae107c61a
  • Simplify handling of remote-qual pass-forward in postgres_fdw. Commit 0bf3ae88a encountered a need to pass the finally chosen remote qual conditions forward from postgresGetForeignPlan to postgresPlanDirectModify. It solved that by sticking them into the plan node's fdw_private list, which in hindsight was a pretty bad idea. In the first place, there's no use for those qual trees either in EXPLAIN or execution; indeed they could never safely be used for any post-planning purposes, because they would not get processed by setrefs.c. So they're just dead weight to carry around in the finished plan tree, plus being an attractive nuisance for somebody who might get the idea that they could be used that way. Secondly, because those qual trees (sometimes) contained RestrictInfos, they created a plan-transmission hazard for parallel query, which is how come we noticed a problem. We dealt with that symptom in commit 28b047875, but really a more straightforward and more efficient fix is to pass the data through in a new field of struct PgFdwRelationInfo. So do it that way. (There's no need to revert 28b047875, as it has sufficient reason to live anyway.) Per fuzz testing by Andreas Seltenreich. Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/88e902b769e180a232013e265ff9fd582dde125b
  • Remove bogus redefinition of _MSC_VER. Commit a4777f355 was a shade too mechanical: we don't want to override MSVC's own definition of _MSC_VER, as that breaks tests on its numerical value. Per buildfarm. http://git.postgresql.org/pg/commitdiff/587d62d8562d658a2a9be60bc4574b6f9e592cb1
  • Mark finished Plan nodes with parallel_safe flags. We'd managed to avoid doing this so far, but it seems pretty obvious that it would be forced on us some day, and this is much the cleanest way of approaching the open problem that parallel-unsafe subplans are being transmitted to parallel workers. Anyway there's no space cost due to alignment considerations, and the time cost is pretty minimal since we're just copying the flag from the corresponding Path node. (At least in most cases ... some of the klugier spots in createplan.c have to work a bit harder.) In principle we could perhaps get rid of SubPlan.parallel_safe, but I thought it better to keep that in case there are reasons to consider a SubPlan unsafe even when its child plan is parallel-safe. This patch doesn't actually do anything with the new flags, but I thought I'd commit it separately anyway. Note: although this touches outfuncs/readfuncs, there's no need for a catversion bump because Plan trees aren't stored on disk. Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/003d80f3dfadd57c6aac8480436ff53ee2c978bd
  • Avoid transferring parallel-unsafe subplans to parallel workers. Commit 5e6d8d2bb allowed parallel workers to execute parallel-safe subplans, but it transmitted the query's entire list of subplans to the worker(s). Since execMain.c blindly does ExecInitNode and later ExecEndNode on every list element, this resulted in parallel-unsafe plan nodes nonetheless getting started up and shut down in parallel workers. That seems mostly harmless as far as core plan node types go (but maybe not so much for Gather?). But it resulted in postgres_fdw opening and then closing extra remote connections, and it's likely that other non-parallel-safe FDWs or custom scan providers would have worse reactions. To fix, just make ExecSerializePlan replace parallel-unsafe subplans with NULLs in the cut-down plan tree that it transmits to workers. This relies on ExecInitNode and ExecEndNode to do nothing on NULL input, but they do anyway. If anything else is touching the dropped subplans in a parallel worker, that would be a bug to be fixed. (This thus provides a strong guarantee that we won't try to do something with a parallel-unsafe subplan in a worker.) This is, I think, the last fix directly occasioned by Andreas Seltenreich's bug report of a few days ago. Tom Lane and Amit Kapila Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/16ebab68862bb5d3595b8c8df083f650d9d7cd20
  • Speed up hash_index regression test. Commit f5ab0a14e made this test take substantially longer than it used to. With a bit more care, we can get the runtime back down while achieving the same, or even a bit better, code coverage. Mithun Cy Discussion: https://postgr.es/m/CAD__Ouh-qaEb+rD7Uy-4g3xQYOrhPzHs-a_TrFAjiQ5azAW5+w@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4a8bc39b08aa83694f22ea56a8626e91e28a0227
  • Move bootstrap-time lookup of regproc OIDs into genbki.pl. Formerly, the bootstrap backend looked up the OIDs corresponding to names in regproc catalog entries using brute-force searches of pg_proc. It was somewhat remarkable that that worked at all, since it was used while populating other pretty-fundamental catalogs like pg_operator. And it was also quite slow, and getting slower as pg_proc gets bigger. This patch moves the lookup work into genbki.pl, so that the values in postgres.bki for regproc columns are always numeric OIDs, an option that regprocin() already supported. Perl isn't the world's speediest language, so this about doubles the time needed to run genbki.pl (from 0.3 to 0.6 sec on my machine). But we only do that at most once per build. The time needed to run initdb drops significantly --- on my machine, initdb --no-sync goes from 1.8 to 1.3 seconds. So this is a small net win even for just one initdb per build, and it becomes quite a nice win for test sequences requiring many initdb runs. Strip out the now-dead code for brute-force catalog searching in regprocin. We'd also cargo-culted similar logic into regoperin and some (not all) of the other reg*in functions. That is all dead code too since we currently have no need to load such values during bootstrap. I removed it all, reasoning that if we ever need such functionality it'd be much better to do it in a similar way to this patch. There might be some simplifications possible in the backend now that regprocin doesn't require doing catalog reads so early in bootstrap. I've not looked into that, though. Andreas Karlsson, with some small adjustments by me Discussion: https://postgr.es/m/30896.1492006367@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/5e39f06cfe65acbecedf42a660f577c3fca47bcc
  • Fix regexport.c to behave sanely with lookaround constraints. regexport.c thought it could just ignore LACON arcs, but the correct behavior is to treat them as satisfiable while consuming zero input (rather reminiscently of commit 9f1e642d5). Otherwise, the emitted simplified-NFA representation may contain no paths leading from initial to final state, which unsurprisingly confuses pg_trgm, as seen in bug #14623 from Jeff Janes. Since regexport's output representation has no concept of an arc that consumes zero input, recurse internally to find the next normal arc(s) after any LACON transitions. We'd be forced into changing that representation if a LACON could be the last arc reaching the final state, but fortunately the regex library never builds NFAs with such a configuration, so there always is a next normal arc. Back-patch to 9.3 where this logic was introduced. Discussion: https://postgr.es/m/20170413180503.25948.94871@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/6cfaffc0ddc73dab6857c094f98b28761898cc6d
  • Further fix pg_trgm's extraction of trigrams from regular expressions. Commit 9e43e8714 turns out to have been insufficient: not only is it necessary to track tentative parent links while considering a set of arc removals, but it's necessary to track tentative flag additions as well. This is because we always merge arc target states into arc source states; therefore, when considering a merge of the final state with some other, it is the other state that will acquire a new TSTATE_FIN bit. If there's another arc for the same color trigram that would cause merging of that state with the initial state, we failed to recognize the problem. The test cases for the prior commit evidently only exercised situations where a tentative merge with the initial state occurs before one with the final state. If it goes the other way around, we'll happily merge the initial and final states, either producing a broken final graph that would never match anything, or triggering the Assert added by the prior commit. It's tempting to consider switching the merge direction when the merge involves the final state, but I lack the time to analyze that idea in detail. Instead just keep track of the flag changes that would result from proposed merges, in the same way that the prior commit tracked proposed parent links. Along the way, add some more debugging support, because I'm not entirely confident that this is the last bug here. And tweak matters so that the transformed.dot file uses small integers rather than pointer values to identify states; that makes it more readable if you're just eyeballing it rather than fooling with Graphviz. And rename a couple of identically named struct fields to reduce confusion. Per report from Corey Csuhta. Add a test case based on his example. (Note: this case does not trigger the bug under 9.3, apparently because its different measurement of costs causes it to stop merging states before it hits the failure. I spent some time trying to find a variant that would fail in 9.3, without success; but I'm sure such cases exist.) Like the previous patch, back-patch to 9.3 where this code was added. Report: https://postgr.es/m/E2B01A4B-4530-406B-8D17-2F67CF9A16BA@csuhta.com http://git.postgresql.org/pg/commitdiff/1dffabed49054a81b6005a363ab2da4aee0aab9e
  • Use one transaction while reading postgres.bki, not one per line. AFAICT, the only actual benefit of closing a bootstrap transaction is to reclaim transient memory. We can do that a lot more cheaply by just doing a MemoryContextReset on a suitable context. This gets the runtime of the "bootstrap" phase of initdb down to the point where, at least by eyeball, it's quite negligible compared to the rest of the phases. Per discussion with Andres Freund. Discussion: https://postgr.es/m/9244.1492106743@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/85a0781334a204c15c9c6ea9d3e6c75334c2beb6
  • Avoid passing function pointers across process boundaries. We'd already recognized that we can't pass function pointers across process boundaries for functions in loadable modules, since a shared library could get loaded at different addresses in different processes. But actually the practice doesn't work for functions in the core backend either, if we're using EXEC_BACKEND. This is the cause of recent failures on buildfarm member culicidae. Switch to passing a string function name in all cases. Something like this needs to be back-patched into 9.6, but let's see if the buildfarm likes it first. Petr Jelinek, with a bunch of basically-cosmetic adjustments by me Discussion: https://postgr.es/m/548f9c1d-eafa-e3fa-9da8-f0cc2f654e60@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/32470825d36d99a81347ee36c181d609c952c061
  • More cleanup of manipulations of hash indexes' hasho_flag field. Not much point in defining test macros for the flag bits if we don't use 'em. Amit Kapila http://git.postgresql.org/pg/commitdiff/083dc95a14c05bdaeec3015508ca1d16fc7483b5
  • Clean up manipulations of hash indexes' hasho_flag field. Standardize on testing a hash index page's type by doing (opaque->hasho_flag & LH_PAGE_TYPE) == LH_xxx_PAGE. Various places were taking shortcuts like opaque->hasho_flag & LH_BUCKET_PAGE which while not actually wrong, is still bad practice because it encourages use of opaque->hasho_flag & LH_UNUSED_PAGE which *is* wrong (LH_UNUSED_PAGE == 0, so the above is constant false). hash_xlog.c's hash_mask() contained such an incorrect test. This also ensures that we mask out the additional flag bits that hasho_flag has accreted since 9.6. pgstattuple's pgstat_hash_page(), for one, was failing to do that and was thus actively broken. Also fix assorted comments that hadn't been updated to reflect the extended usage of hasho_flag, and fix some macros that were testing just "(hasho_flag & bit)" to use the less dangerous, project-approved form "((hasho_flag & bit) != 0)". Coverity found the bug in hash_mask(); I noted the one in pgstat_hash_page() through code reading. http://git.postgresql.org/pg/commitdiff/2040bb4a0b50ef0434a1a723f00d040ab4f1c06f
  • Fix erroneous cross-reference in comment. Seems to have been introduced in commit c219d9b0a. I think there indeed was a "tupbasics.h" in some early drafts of that refactoring, but it didn't survive into the committed version. Amit Kapila http://git.postgresql.org/pg/commitdiff/bfba563bc5d80263637a3cfba6a572c20949defe
  • Provide a way to control SysV shmem attach address in EXEC_BACKEND builds. In standard non-Windows builds, there's no particular reason to care what address the kernel chooses to map the shared memory segment at. However, when building with EXEC_BACKEND, there's a risk that the chosen address won't be available in all child processes. Linux with ASLR enabled (which it is by default) seems particularly at risk because it puts shmem segments into the same area where it maps shared libraries. We can work around that by specifying a mapping address that's outside the range where shared libraries could get mapped. On x86_64 Linux, 0x7e0000000000 seems to work well. This is only meant for testing/debugging purposes, so it doesn't seem necessary to go as far as providing a GUC (or any user-visible documentation, though we might change that later). Instead, it's just controlled by setting an environment variable PG_SHMEM_ADDR to the desired attach address. Back-patch to all supported branches, since the point here is to remove intermittent buildfarm failures on EXEC_BACKEND animals. Owners of affected animals will need to add a suitable setting of PG_SHMEM_ADDR to their build_env configuration. Discussion: https://postgr.es/m/7036.1492231361@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/a74740fbd3bb89cd626f6e98417847f696e60bd8
  • Sync addRangeTableEntryForENR() with its peer functions. addRangeTableEntryForENR had a check for pstate != NULL, which Coverity pointed out was rather useless since it'd already dereferenced pstate before that. More to the point, we'd established policy in commit bc93ac12c that we'd require non-NULL pstate for all addRangeTableEntryFor* functions; this test was evidently copied-and-pasted from some older version of one of those functions. Make it look more like the others. In passing, make an elog message look more like the rest of the code, too. Michael Paquier http://git.postgresql.org/pg/commitdiff/a1888b59b511b42290a6fcfa87e35323d128c4f6

Peter Eisentraut pushed:

Robert Haas pushed:

Andres Freund pushed:

Andrew Dunstan pushed:

Michael Meskes pushed:

Magnus Hagander pushed:

  • Remove support for bcc and msvc standalone libpq builds. This removes the support for building just libpq using Borland C++ or Visual C++. This has not worked properly for years, and given the number of complaints it's clearly not worth the maintenance burden. Building libpq using the standard MSVC build system is of course still supported, along with mingw. http://git.postgresql.org/pg/commitdiff/6da56f3f84d430671d5edd8f9336bd744c089e31
  • Remove symbol WIN32_ONLY_COMPILER. This used to mean "Visual C++ except in those parts where Borland C++ was supported where it meant one of those". Now that we don't support Borland C++ anymore, simplify by using _MSC_VER which is the normal way to detect Visual C++. http://git.postgresql.org/pg/commitdiff/a4777f35565b80ae10605d6d417e5d173156f7da
  • Fix reversed check of return value from sync. While at it also update the comments in walmethods.h to make it less likely for mistakes like this to appear in the future (thanks to Tom for improvements to the comments). And finally, in passing change the return type of walmethod.getlasterror to being const, also per suggestion from Tom. http://git.postgresql.org/pg/commitdiff/b935eb7da37254a18c48acc7b07517c8dfbb2339

Bruce Momjian pushed:

Fujii Masao pushed:

Simon Riggs pushed:

Álvaro Herrera pushed:

Correctifs en attente

Michaël Paquier sent in another revision of a patch to use base64-based encoding for stored and server keys in SCRAM verifiers, move the routine to build SCRAM verifier into src/common/, refactor frontend-side random number generation, extend PQencryptPassword with a hashing method, and extend psql's \password and createuser to handle SCRAM.

Michaël Paquier sent in a patch to change pg_hba.conf to refer to SASL, the thing being implemented, instead of SCRAM, one implementation of it.

Ildar Musin sent in a patch to factor out some repetitive code in RI triggers.

Peter Eisentraut sent in two revisions of a patch to fix some warnings generated by gcc7.

David Rowley sent in a patch to allow extended stats on foreign and partitioned tables.

Peter Eisentraut and Bruce Momjian traded patches to fix an issue around HTML rendering in the new documentation toolchain.

Jeff Davis sent in two more revisions of a patch to implement range merge joins.

Alexander Kuzmenkov sent in a patch to implement an index-only count(*) for indexes supporting bitmap scans.

Craig Ringer and Mithun Cy traded patches to make TAP tests run faster.

Kyotaro HORIGUCHI sent in two more revisions of a patch to fix wal_level_minimal.

Amit Langote sent in three more revisions of a patch to fix how pg_dump emits partition attributes.

Masahiko Sawada sent in another revision of a patch to make pg_stat_replication.sync priority report NULL if in quorum-based sync replication, and a different one to change the comments and documentation.

Takayuki Tsunakawa sent in another revision of a patch to make huge pages work on Windows.

Craig Ringer sent in a patch to fix undefined var warnings in PostgresNode with timeout.

Dmitry Ivanov sent in a patch to fix an issue with the custom scan API.

Ashutosh Bapat sent in three more revisions of a patch to fix reporting of violation in ExecConstraints.

Thomas Munro sent in another revision of a patch to implement [[Parallel] Shared] Hash.

Yorick Peterse sent in another revision of a patch to document hot standby config order mentioning all standby servers.

Michaël Paquier sent in another revision of a patch to reimplement pg_upgrade's test as a TAP test.

Pierre Ducroquet sent in a patch to check the results of atoi in pg_basebackup argument parsing.

Amit Langote sent in a patch to ensure that duplicate addition to publication is a no-op.

Petr Jelínek sent in another revision of a patch to set range table for CopyFrom in tablesync.

Petr Jelínek sent in another revision of a patch to reserve global xmin for create slot snasphot export, ensure that on-disk snapshots are not used for snapshot export in logical decoding, prevent snapshot builder xmin from going backwards, fix xl_running_xacts usage in snapshot builder, and skip unnecessary snapshot builds.

Amit Langote sent in a patch to fix a typo in partition.c.

Jeff Janes and Pavan Deolasee traded patches to fix an issue that manifested as PANIC in pg_commit_ts slru after crashes.

Masahiko Sawada sent in a patch to fix a comment typo in xlogutils.c.

Jeff Janes sent in a patch to fix an issue that manifested as failed recovery with the new faster 2PC code.

par chl le lundi 17 avril 2017 à 21h06

jeudi 13 avril 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 9 avril 2017

[ndt: Meetup à Lyon le mercredi 26 avril : https://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170409231442.GA4338@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Document psql's behavior of recalling the previously executed query. Various psql slash commands that normally act on the current query buffer will automatically recall and re-use the most recently executed SQL command instead, if the current query buffer is empty. Although this behavior is ancient (dating apparently to commit 77a472993), it was documented nowhere in the psql reference page. For that matter, we'd never bothered to define the concept of "current query buffer" explicitly. Fix that. Do some wordsmithing on relevant command descriptions to improve clarity and consistency. Discussion: https://postgr.es/m/9b4ea968-753f-4b5f-b46c-d7d3bf7c8f90@manitou-mail.org http://git.postgresql.org/pg/commitdiff/68dba97a4dea5c5c915e31978a475107c17c458d
  • Doc: clarify behavior of OT_WHOLE_LINE and OT_FILEPIPE psql slash commands. This is another bit of ancient behavior that was documented poorly (in a couple of cases) or not at all (in several others). Discussion: https://postgr.es/m/9b4ea968-753f-4b5f-b46c-d7d3bf7c8f90@manitou-mail.org http://git.postgresql.org/pg/commitdiff/ffac5998b4c18920f86d80f1bddbde9ebcf0a314
  • Remove reinvention of stringify macro. We already have CppAsString2, there's no need for the MSVC support to re-invent a macro to do that (and especially not to inject it in as ugly a way as this). Discussion: https://postgr.es/m/CADkLM=c+hm2rc0tkKgC-ZgrLttHT2KkfppE+BC-=i-xj+7V-TQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/cd6baed78150e107eb858dbd53ddf900dd429f50
  • Fix integer-overflow problems in interval comparison. When using integer timestamps, the interval-comparison functions tried to compute the overall magnitude of an interval as an int64 number of microseconds. As reported by Frazer McLean, this overflows for intervals exceeding about 296000 years, which is bad since we nominally allow intervals many times larger than that. That results in wrong comparison results, and possibly in corrupted btree indexes for columns containing such large interval values. To fix, compute the magnitude as int128 instead. Although some compilers have native support for int128 calculations, many don't, so create our own support functions that can do 128-bit addition and multiplication if the compiler support isn't there. These support functions are designed with an eye to allowing the int128 code paths in numeric.c to be rewritten for use on all platforms, although this patch doesn't do that, or even provide all the int128 primitives that will be needed for it. Back-patch as far as 9.4. Earlier releases did not guard against overflow of interval values at all (commit 146604ec4 fixed that), so it seems not very exciting to worry about overly-large intervals for them. Before 9.6, we did not assume that unreferenced "static inline" functions would not draw compiler warnings, so omit functions not directly referenced by timestamp.c, the only present consumer of int128.h. (We could have omitted these functions in HEAD too, but since they were written and debugged on the way to the present patch, and they look likely to be needed by numeric.c, let's keep them in HEAD.) I did not bother to try to prevent such warnings in a --disable-integer-datetimes build, though. Before 9.5, configure will never define HAVE_INT128, so the part of int128.h that exploits a native int128 implementation is dead code in the 9.4 branch. I didn't bother to remove it, thinking that keeping the file looking similar in different branches is more useful. In HEAD only, add a simple test harness for int128.h in src/tools/. In back branches, this does not change the float-timestamps code path. That's not subject to the same kind of overflow risk, since it computes the interval magnitude as float8. (No doubt, when this code was originally written, overflow was disregarded for exactly that reason.) There is a precision hazard instead :-(, but we'll avert our eyes from that question, since no complaints have been reported and that code's deprecated anyway. Kyotaro Horiguchi and Tom Lane Discussion: https://postgr.es/m/1490104629.422698.918452336.26FA96B7@webmail.messagingengine.com http://git.postgresql.org/pg/commitdiff/df1a699e5ba3232f373790b2c9485ddf720c4a70
  • Clean up psql/describe.c's messy query for extended stats. Remove unnecessary casts, safely schema-qualify the ones that remain, lose an unnecessary level of sub-SELECT, reformat for tidiness. http://git.postgresql.org/pg/commitdiff/20c95f27e736837b4af6bef998cb9408d1ad902e
  • Clean up after insufficiently-researched optimization of tuple conversions. tupconvert.c's functions formerly considered that an explicit tuple conversion was necessary if the input and output tupdescs contained different type OIDs. The point of that was to make sure that a composite datum resulting from the conversion would contain the destination rowtype OID in its composite-datum header. However, commit 3838074f8 entirely misunderstood what that check was for, thinking that it had something to do with presence or absence of an OID column within the tuple. Removal of the check broke the no-op conversion path in ExecEvalConvertRowtype, as reported by Ashutosh Bapat. It turns out that of the dozen or so call sites for tupconvert.c functions, ExecEvalConvertRowtype is the only one that cares about the composite-datum header fields in the output tuple. In all the rest, we'd much rather avoid an unnecessary conversion whenever the tuples are physically compatible. Moreover, the comments in tupconvert.c only promise physical compatibility not a metadata match. So, let's accept the removal of the guarantee about the output tuple's rowtype marking, recognizing that this is a API change that could conceivably break third-party callers of tupconvert.c. (So, let's remember to mention it in the v10 release notes.) However, commit 3838074f8 did have a bit of a point here, in that two tuples mustn't be considered physically compatible if one has HEAP_HASOID set and the other doesn't. (Some of the callers of tupconvert.c might not really care about that, but we can't assume it in general.) The previous check accidentally covered that issue, because no RECORD types ever have OIDs, while if two tupdescs have the same named composite type OID then, a fortiori, they have the same tdhasoid setting. If we're removing the type OID match check then we'd better include tdhasoid match as part of the physical compatibility check. Without that hack in tupconvert.c, we need ExecEvalConvertRowtype to take responsibility for inserting the correct rowtype OID label whenever tupconvert.c decides it need not do anything. This is easily done with heap_copy_tuple_as_datum, which will be considerably faster than a tuple disassembly and reassembly anyway; so from a performance standpoint this change is a win all around compared to what happened in earlier branches. It just means a couple more lines of code in ExecEvalConvertRowtype. Ashutosh Bapat and Tom Lane Discussion: https://postgr.es/m/CAFjFpRfvHABV6+oVvGcshF8rHn+1LfRUhj7Jz1CDZ4gPUwehBg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/3f902354b08ac788600f0ae54fcbfc1d4e3ea765
  • Fix planner error (or assert trap) with nested set operations. As reported by Sean Johnston in bug #14614, since 9.6 the planner can fail due to trying to look up the referent of a Var with varno 0. This happens because we generate such Vars in generate_append_tlist, for lack of any better way to describe the output of a SetOp node. In typical situations nothing really cares about that, but given nested set-operation queries we will call estimate_num_groups on the output of the subquery, and that wants to know what a Var actually refers to. That logic used to look at subquery->targetList, but in commit 3fc6e2d7f I'd switched it to look at subroot->processed_tlist, ie the actual output of the subquery plan not the parser's idea of the result. It seemed like a good idea at the time :-(. As a band-aid fix, change it back. Really we ought to have an honest way of naming the outputs of SetOp steps, which suggests that it'd be a good idea for the parser to emit an RTE corresponding to each one. But that's a task for another day, and it certainly wouldn't yield a back-patchable fix. Report: https://postgr.es/m/20170407115808.25934.51866@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/89deca582a345b9c423bed8ebcf24b6ee81a9953
  • Ensure that ExecPrepareExprList's result is all in one memory context. Noted by Amit Langote. Discussion: https://postgr.es/m/aad31672-4983-d95d-d24e-6b42fee9b985@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/dbb2a931478a397a2b655eb77e8be8c1ca136f63
  • Optimize joins when the inner relation can be proven unique. If there can certainly be no more than one matching inner row for a given outer row, then the executor can move on to the next outer row as soon as it's found one match; there's no need to continue scanning the inner relation for this outer row. This saves useless scanning in nestloop and hash joins. In merge joins, it offers the opportunity to skip mark/restore processing, because we know we have not advanced past the first possible match for the next outer row. Of course, the devil is in the details: the proof of uniqueness must depend only on joinquals (not otherquals), and if we want to skip mergejoin mark/restore then it must depend only on merge clauses. To avoid adding more planning overhead than absolutely necessary, the present patch errs in the conservative direction: there are cases where inner_unique or skip_mark_restore processing could be used, but it will not do so because it's not sure that the uniqueness proof depended only on "safe" clauses. This could be improved later. David Rowley, reviewed and rather heavily editorialized on by me Discussion: https://postgr.es/m/CAApHDvqF6Sw-TK98bW48TdtFJ+3a7D2mFyZ7++=D-RyPsL76gw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/9c7f5229ad68d7e0e4dd149e3f80257893e404d4
  • Add newly-symlinked files to "make clean" target. Oversight in 60f11b87a. http://git.postgresql.org/pg/commitdiff/aba696d1af9a267eee85d69845c3cdeccf788525
  • Clean up bugs in clause_selectivity() cleanup. Commit ac2b09508 was not terribly carefully reviewed. Band-aid it to not fail on non-RestrictInfo input, per report from Andreas Seltenreich. Also make it do something more reasonable with variable-free clauses, and improve nearby comments. Discussion: https://postgr.es/m/87inmf5rdx.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/eef8c0069e4d5eea2e52965ce3eb018b5a594fd6

Peter Eisentraut pushed:

Andrew Gierth pushed:

Robert Haas pushed:

Stephen Frost pushed:

Simon Riggs pushed:

Andrew Dunstan pushed:

Andres Freund pushed:

Kevin Grittner pushed:

  • Follow-on cleanup for the transition table patch. Commit 59702716 added transition table support to PL/pgsql so that SQL queries in trigger functions could access those transient tables. In order to provide the same level of support for PL/perl, PL/python and PL/tcl, refactor the relevant code into a new function SPI_register_trigger_data. Call the new function in the trigger handler of all four PLs, and document it as a public SPI function so that authors of out-of-tree PLs can do the same. Also get rid of a second QueryEnvironment object that was maintained by PL/pgsql. That was previously used to deal with cursors, but the same approach wasn't appropriate for PLs that are less tangled up with core code. Instead, have SPI_cursor_open install the connection's current QueryEnvironment, as already happens for SPI_execute_plan. While in the docs, remove the note that transition tables were only supported in C and PL/pgSQL triggers, and correct some ommissions. Thomas Munro with some work by Kevin Grittner (mostly docs) http://git.postgresql.org/pg/commitdiff/5ebeb579b9b281dba5f8415b2fbda86fdae7b366
  • Add isolation test for SERIALIZABLE READ ONLY DEFERRABLE. This improves code coverage and lays a foundation for testing similar issues in a distributed environment. Author: Thomas Munro <thomas.munro@enterprisedb.com> Reviewed-by: Michael Paquier <michael.paquier@gmail.com> http://git.postgresql.org/pg/commitdiff/4deb413813f619b3e859abf435b61efc09cafe09
  • Fix the RTE_NAMEDTUPLESTORE case in get_rte_attribute_is_dropped(). Problems pointed out by Andres Freund and Thomas Munro. http://git.postgresql.org/pg/commitdiff/255efa241f460ee4f4c4c98c8cdd7457807f3af9
  • Add GUCs for predicate lock promotion thresholds. Defaults match the fixed behavior of prior releases, but now DBAs have better options to tune serializable workloads. It might be nice to be able to set this per relation, but that part will need to wait for another release. Author: Dagfinn Ilmari Mannsåker http://git.postgresql.org/pg/commitdiff/c63172d60f242ad3581c83723a5b315bbe547a0e

Heikki Linnakangas pushed:

Álvaro Herrera pushed:

Joe Conway pushed:

Magnus Hagander pushed:

Correctifs en attente

Vaishnavi Prabakaran sent in two more revisions of a patch to add batch/pipelining support to libpq.

Craig Ringer sent in five more revisions of a patch to add logical decoding on standby.

Ashutosh Bapat and Amit Langote traded patches to document some intricacies of the relationship between declarative partitions and foreign tables.

Takayuki Tsunakawa sent in three more revisions of a patch to support huge pages on Windows.

Amit Khandekar sent in two more revisions of a patch to enable UPDATEs on partitioned tables which would cause rows to move among partitions to work.

Rushabh Lathia sent in a patch to add coverage tests for gather merge.

Amit Langote sent in a patch to enable ON CONFLICT DO NOTHING on partitioned tables.

Haribabu Kommi sent in another revision of a patch to refactor handling of database attributes between pg_dump and pg_dumpall.

Alexander Korotkov sent in another revision of a patch to implement incremental sort.

Daniel Gustafsson sent in a patch to use strcmp() instead of pg_strcasecmp() for identifier matching.

Kyotaro HORIGUCHI sent in a patch to distinguish "aggressive" VACUUMs from non- in logs.

Vinayak Pokale sent in another revision of a patch to add an ANALYZE progress checker.

Tatsuo Ishii and Andres Freund traded patches to rearm statement_timeout after each executed query.

Michaël Paquier sent in two revisions of a patch to rewrite the test of pg_upgrade as a TAP test.

Kuntal Ghosh sent in two revisions of a patch to fix parallel worker counts after a crash.

Amit Langote sent in a patch to remove pg_stat_progress_vacuum from Table 28.2.

David Rowley sent in three more revisions of a patch to make clausesel smarter.

Antonin Houska sent in another revision of a patch to implement aggregation pushdown.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous execution.

Maksim Milyutin sent in a patch to create a "local" index for partitioned tables.

Fabien COELHO sent in a patch to add a special variable in psql to reflect the last query status.

Ashutosh Bapat sent in two more revisions of a patch to implement partition-wise joins for declaratively partitioned tables.

Rahila Syed sent in three more revisions of a patch to add support for a default partition in declarative partitions.

Craig Ringer and Jim Nasby traded patches to add SPI_execute_callback and documentation for same, then use it in PL/PythonU to speed up SPI results.

Dmitry Dolgov sent in two more revisions of a patch to implement generic type subscripting.

Amit Khandekar sent in another revision of a patch to implement parallel append.

Etsuro Fujita sent in another revision of a patch to fix a bug in the PostgreSQL FDW.

Pavan Deolasee sent in two more revisions of a patch to implement WARM.

Mark Dilger sent in a patch to create a generic PG_GETARG_*_P infrastructure.

Robert Haas sent in another revision of a patch to add pgstathashindex() to get hash index table statistics.

Etsuro Fujita sent in another revision of a patch to support parameterized foreign joins.

Fabien COELHO sent in three more revisions of a patch to enable pgbench to store select results into variables.

Jeff Davis sent in a patch to implement range merge join.

Ashutosh Bapat sent in a patch to implement constraint exclusion for partitioned tables.

Kyotaro HORIGUCHI sent in a patch to ensure that enabling subscription starts a worker.

Michaël Paquier sent in a patch to use base64-based encoding for stored and server keys in SCRAM verifiers, move the routine to build the SCRAM verifier into src/common/, refactor frontend-side random number generation, extend PQencryptPassword with a hashing method, and extend psql's \password and createuser to handle SCRAM verifier creation.

Tatsuro Yamada sent in a patch to make postgresGetForeignPlan use foreignrel instead of baserel.

Thomas Munro sent in a patch to implement hash tables in dynamic shared memory.

Thomas Munro sent in a PoC patch to enable sharing record typmods between backends.

Vitaly Burovoy sent in a patch to implement SET IDENTITY ... IF NOT EXISTS.

Kyotaro HORIGUCHI sent in a patch to fix distclean of src/backend/utils/mb/Unicode.

Yorick Peterse sent in two revisions of a patch to update the hot-standby documentation (in the high availability section) so it explicitly mentions that certain settings need to be applied to servers in a particular order.

Beena Emerson sent in a patch to update the initdb regression tests to include increasing the default WAL segment size.

Aleksander Alekseev sent in another revision of a patch to remove an unused argument in btree_xlog_split.

Tatsuo Ishii sent in a patch to document the fact that pg_export_snapshot() cannot be used during recovery (i.e. on standby servers).

Aleksander Alekseev sent in a patch to warn users about duplicate configuration parameters in postgresql.conf.

Álvaro Herrera sent in another revision of a patch to fix wal_level_minimal.

Thomas Munro sent in a patch to initialise a freed segment counter in dsa.c.

Claudio Freire sent in another revision of a patch to allow VACUUM to use more than 1GB work mem in, and make it free dead tuples array as early as possible.

Thomas Munro sent in a patch to fix and document some snapshot issues.

Thomas Munro sent in a patch to add a pg_waiting_for_safe_snapshot() function.

Tom Lane and Mark Dilger traded patches to use IsA checks for bitmapsets.

Michael Banck sent in another revision of a patch to add an option to create a replication slot in pg_basebackup if not yet present.

par chl le jeudi 13 avril 2017 à 23h56

mardi 4 avril 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 2 avril 2017

PGDay UK 2017 - Inscriptions et appel à conférenciers lancés : http://www.pgconf.uk

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170402225015.GC10244@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Fix unportable disregard of alignment requirements in RADIUS code. The compiler is entitled to store a char[] local variable with no particular alignment requirement. Our RADIUS code cavalierly took such a local variable and cast its address to a struct type that does have alignment requirements. On an alignment-picky machine this would lead to bus errors. To fix, declare the local variable honestly, and then cast its address to char * for use in the I/O calls. Given the lack of field complaints, there must be very few if any people affected; but nonetheless this is a clear portability issue, so back-patch to all supported branches. Noted while looking at a Coverity complaint in the same code. http://git.postgresql.org/pg/commitdiff/4c051c41d63e8462971c13956a38030de9c35c51
  • Fix some minor resource leaks in PerformRadiusTransaction(). Failure to free serveraddrs pointed out by Coverity, failure to close socket noted by code-reading. These bugs seem to be quite old, but given the low probability of taking these error-exit paths and the minimal consequences of the leaks (since the process would presumably exit shortly anyway), it doesn't seem worth back-patching. Michael Paquier and Tom Lane http://git.postgresql.org/pg/commitdiff/7cbd944662854a0a5264895bcba3ce7f9bfd1c1f
  • Fix typos in logical replication support for initial data copy. Fix an incorrect assert condition (noted by Coverity), and spell the new name of the function correctly. Typos introduced in commit 7c4f52409. Michael Paquier http://git.postgresql.org/pg/commitdiff/5459cfd3ad52b87a1e2ed293ae55e733c6964715
  • Use ExecPrepareExpr in place of ExecPrepareCheck where appropriate. Change one more place where ExecInitCheck/ExecPrepareCheck's insistence on getting implicit-AND-format quals wasn't really helpful, because the caller had to do make_ands_implicit() for no reason that it cared about. Using ExecPrepareExpr directly simplifies the code and saves cycles. The only remaining use of these functions is to process resultRelInfo->ri_PartitionCheck quals. However, implicit-AND format does seem to be what we want for that, so leave it alone. http://git.postgresql.org/pg/commitdiff/9b95f2fa1e2684fa209a3594db2254b8841bf380
  • Improve performance of ExecEvalWholeRowVar. In commit b8d7f053c, we needed to fix ExecEvalWholeRowVar to not change the state of the slot it's copying. The initial quick hack at that required two rounds of tuple construction, which is not very nice. To fix, add another primitive to tuptoaster.c that does precisely what we need. (I initially tried to do this by refactoring one of the existing functions into two pieces; but it looked like that might hurt performance for the existing case, and the amount of code that could be shared is not very large, so I gave up on that.) Discussion: https://postgr.es/m/26088.1490315792@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/2f0903ea196503fc8af373a9de46b1e01a23508c
  • Show ignored constants as "$N" rather than "?" in pg_stat_statements. The trouble with the original choice here is that "?" is a valid (and indeed used) operator name, so that you could end up with ambiguous statement texts like "SELECT ? ? ?". With this patch, you instead see "SELECT $1 ? $2", which seems significantly more readable. The numbers used for this purpose begin after the last actual $N parameter in the particular query. The conflict with external parameters has its own potential for confusion of course, but it was agreed to be an improvement over the previous behavior. Lukas Fittl Discussion: https://postgr.es/m/CAP53PkxeaCuwYmF-A4J5z2-qk5fYFo5_NH3gpXGJJBxv1DMwEw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/a6f22e83562d8b78293229587cd3d9430d16d466
  • Suppress implicit-conversion warnings seen with newer clang versions. We were assigning values near 255 through "char *" pointers. On machines where char is signed, that's not entirely kosher, and it's reasonable for compilers to warn about it. A better solution would be to change the pointer type to "unsigned char *", but that would be vastly more invasive. For the moment, let's just apply this simple backpatchable solution. Aleksander Alekseev Discussion: https://postgr.es/m/20170220141239.GD12278@e733.localdomain Discussion: https://postgr.es/m/2839.1490714708@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/8cfeaecfc76a7366b336272bc76e96e09281b133
  • Make new expression eval code reject references to dropped columns. Formerly, a Var referencing an already-dropped column was allowed and would always produce a NULL value. However, that behavior was implemented in slot_getattr which the new expression code doesn't use; thus there is now a risk of returning theoretically-deleted data. We had regression test cases that purported to exercise this, but they failed to expose any problem, apparently because plpgsql filters the dropped column and produces an output tuple that has a NULL there already. Ideally the DROP or ALTER attempt in these test cases would get rejected due to dependency checks; but until that happens, let's modify the behavior so that we fail the query during executor start. This was already true for the related case of a column having changed type underneath us, and there's no obvious reason why we need to be laxer for dropped columns. In passing, adjust the error messages in CheckVarSlotCompatibility to include the composite type name. In the cases shown in the regression tests this is always just "record", but it should be more useful in actual stale-plan cases, where the slot tupdesc would be a table's tupdesc directly. Discussion: https://postgr.es/m/16803.1490723570@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/2c4debbd0f018aa7322b622c88424a7f68d3081d
  • Support \if ... \elif ... \else ... \endif in psql scripting. This patch adds nestable conditional blocks to psql. The control structure feature per se is complete, but the boolean expressions understood by \if and \elif are pretty primitive; basically, after variable substitution and backtick expansion, the result has to be "true" or "false" or one of the other standard spellings of a boolean value. But that's enough for many purposes, since you can always do the heavy lifting on the server side; and we can extend it later. Along the way, pay down some of the technical debt that had built up around psql/command.c: * Refactor exec_command() into a function per command, instead of being a 1500-line monstrosity. This makes the file noticeably longer because of repetitive function header/trailer overhead, but it seems much more readable. * Teach psql_get_variable() and psqlscanslash.l to suppress variable substitution and backtick expansion on the basis of the conditional stack state, thereby allowing removal of the OT_NO_EVAL kluge. * Fix the no-doubt-once-expedient hack of sometimes silently substituting mainloop.c's previous_buf for query_buf when calling HandleSlashCmds. (It's a bit remarkable that commands like \r worked at all with that.) Recall of a previous query is now done explicitly in the slash commands where that should happen. Corey Huinker, reviewed by Fabien Coelho, further hacking by me Discussion: https://postgr.es/m/CADkLM=c94OSRTnat=LX0ivNq4pxDNeoomFfYvBKM5N_xfmLtAA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/e984ef5861df4bc9733b36271d05763e82de7c04
  • Fix broken markup. Per buildfarm. http://git.postgresql.org/pg/commitdiff/ab1e644005b6ef77dada51188d7b92905e2444d7
  • For foreign keys, check REFERENCES privilege only on the referenced table. We were requiring that the user have REFERENCES permission on both the referenced and referencing tables --- but this doesn't seem to have any support in the SQL standard, which says only that you need REFERENCES permission on the referenced table. And ALTER TABLE ADD FOREIGN KEY has already checked that you own the referencing table, so the check could only fail if a table owner has revoked his own REFERENCES permission. Moreover, the symmetric interpretation of this permission is unintuitive and confusing, as per complaint from Paul Jungwirth. So let's drop the referencing-side check. In passing, do a bit of wordsmithing on the GRANT reference page so that all the privilege types are described in similar fashion. Discussion: https://postgr.es/m/8940.1490906755@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/64d4da511c012faff8ac309595620938a43c6817
  • Fix unstable regression test result. Commit e306df7f9 added a test case that depends on "the" being a stop word, which it is not in non-English locales. Since the point of the test is to check stopword behavior, fix by forcibly selecting the 'english' configuration. Per buildfarm. http://git.postgresql.org/pg/commitdiff/f1a285e21111f4d4d0c3f428ce2b3ae9e7f162c2
  • Fix unstable regression test result. Whoops, missed that same test was made for json as well as jsonb. http://git.postgresql.org/pg/commitdiff/c281cd5fe178c946dc23eae4d4642be5ddbe3eb4
  • Allow psql variable substitution to occur in backtick command strings. Previously, text between backquotes in a psql metacommand's arguments was always passed to the shell literally. That considerably hobbles the usefulness of the feature for scripting, so we'd foreseen for a long time that we'd someday want to allow substitution of psql variables into the shell command. IMO the addition of \if metacommands has brought us to that point, since \if can greatly benefit from some sort of client-side expression evaluation capability, and psql itself is not going to grow any such thing in time for v10. Hence, this patch. It allows :VARIABLE to be replaced by the exact contents of the named variable, while :'VARIABLE' is replaced by the variable's contents suitably quoted to become a single shell-command argument. (The quoting rules for that are different from those for SQL literals, so this is a bit of an abuse of the :'VARIABLE' notation, but I doubt anyone will be confused.) As with other situations in psql, no substitution occurs if the word following a colon is not a known variable name. That limits the risk of compatibility problems for existing psql scripts; but the risk isn't zero, so this needs to be called out in the v10 release notes. Discussion: https://postgr.es/m/9561.1490895211@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/f833c847b8fa4782efab45c8371d3cee64292d9b
  • Fix behavior of psql's \p to agree with \g, \w, etc. In commit e984ef586 I (tgl) simplified the behavior of \p to just print the current query buffer; but Daniel Vérité points out that this made it inconsistent with the behavior of \g and \w. It should print the same thing \g would execute. Fix that, and improve related comments. Daniel Vérité Discussion: https://postgr.es/m/9b4ea968-753f-4b5f-b46c-d7d3bf7c8f90@manitou-mail.org http://git.postgresql.org/pg/commitdiff/5dbc5da1187c1ddb6e091047194d364337ebf232

Peter Eisentraut pushed:

Robert Haas pushed:

Andrew Gierth pushed:

Teodor Sigaev pushed:

Ãlvaro Herrera pushed:

  • Rework the stats_ext test. As suggested by Tom Lane, avoid printing specific estimated cost values, because they vary across architectures; instead, verify plan shapes (in this case, HashAggregate vs. GroupAggregate), as we do in other planner tests. We can now remove expected/stats_ext_1.out. Author: Tomas Vondra http://git.postgresql.org/pg/commitdiff/bed9ef5a16239d91d97a1fa2efd9309c3cbbc4b2
  • Fix a couple of problems in pg_get_statisticsextdef. There was a thinko whereby we tested the wrong tuple after fetching it from cache; avoid that by using generate_relation_name instead, which is simpler. Also, the statistics name was not qualified, so add that. (It could be argued that qualification should be conditional on the schema not being on search path. We can add that later, but at least this form is correct.) Author: David Rowley, Ãlvaro Herrera Discussion: https://postgr.es/m/CAKJS1f8RjLeVZJ2+93pdQGuZJeBF-ifsHaFMR-q-6-Z0qxA8cA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/2c3e47527a6f53cd1d98887fdb9e770c118954ca
  • Fix thinko in estimate_num_groups. The code for the reworked n-distinct estimation on commit 7b504eb282 was written differently in a previous version of the patch, prior to commit; on rewriting it, we missed updating an initializer. This caused the code to (mistakenly) apply a fudge factor even in the case where a single value is applied, leading to incorrect results. This means that the 'relvarcount' variable name is now wrong. Add a comment to try and make the situation clearer, and remove an incorrect comment I added. Problem noticed, and code patch, by Tomas Vondra. Additional commentary by Ãlvaro. http://git.postgresql.org/pg/commitdiff/1f171a1803c28d3ae24636c9ca3352ec82c39e5f
  • Fix uninitialized memory propagation mistakes. Valgrind complains that some uninitialized bytes are being passed around by the extended statistics code since commit 7b504eb282ca2f, as reported by Andres Freund. Silence it. Tomas Vondra submitted a patch which he verified to fix the complaints in his machine; however I messed with it a bit before pushing, so any remaining problems are likely my (Ãlvaro's) fault. Author: Tomas Vondra Discussion: https://postgr.es/m/20170325211031.4xxoptigqxm2emn2@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/6462238f0d7b7c15eb3f54c2108573cee8fb24ba
  • Remove direct uses of ItemPointer.{ip_blkid,ip_posid}. There are no functional changes here; this simply encapsulates knowledge of the ItemPointerData struct so that a future patch can change things without more breakage. All direct users of ip_blkid and ip_posid are changed to use existing macros ItemPointerGetBlockNumber and ItemPointerGetOffsetNumber respectively. For callers where that's inappropriate (because they Assert that the itempointer is is valid-looking), add ItemPointerGetBlockNumberNoCheck and ItemPointerGetOffsetNumberNoCheck, which lack the assertion but are otherwise identical. Author: Pavan Deolasee Discussion: https://postgr.es/m/CABOikdNnFon4cJiL=h1mZH3bgUeU+sWHuU4Yr8AB=j3A2p1GiA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/ce96ce60ca2293f75f36c3661e4657a3c79ffd61
  • Allow DSM segments to be created as pinned. dsm_create and dsm_attach assumed that a current resource owner was always in place. Exploration with the API show that this is inconvenient: sometimes one must create a dummy resowner, create/attach the DSM, only to pin the mapping later, which is wasteful. Change create/attach so that if there is no current resowner, the dsm is effectively pinned right from the start. Discussion: https://postgr.es/m/20170324232710.32acsfsvjqfgc6ud@alvherre.pgsql Reviewed by Thomas Munro. http://git.postgresql.org/pg/commitdiff/767bc028e5f001351feb498acef9a87c123093d6
  • Simplify check of modified attributes in heap_update. The old coding was getting more complicated as new things were added, and it would be barely tolerable with upcoming WARM updates and other future features such as indirect indexes. The new coding incurs a small performance cost in synthetic benchmark cases, and is barely measurable in normal cases. A much larger benefit is expected from WARM, which could actually bolt its needs on top of the existing coding, but it is much uglier and bug-prone than doing it on this new code. Additional optimization can be applied on top of this, if need be. Reviewed-by: Pavan Deolasee, Amit Kapila, Mithun CY Discussion: https://postgr.es/m/20161228232018.4hc66ndrzpz4g4wn@alvherre.pgsql https://postgr.es/m/CABOikdMJfz69dBNRTOZcB6s5A0tf8OMCyQVYQyR-WFFdoEwKMQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/2fd8685e7fd9fddf16f99de1284a787d29781cc8
  • Fix expected output. Previous commit had a thinko in the expected output for new tests. Per buildfarm http://git.postgresql.org/pg/commitdiff/3a82129a40a3a2987356d4051e017fd456876c8d
  • BRIN de-summarization. When the BRIN summary tuple for a page range becomes too "wide" for the values actually stored in the table (because the tuples that were present originally are no longer present due to updates or deletes), it can be useful to remove the outdated summary tuple, so that a future summarization can install a tighter summary. This commit introduces a SQL-callable interface to do so. Author: Ãlvaro Herrera Reviewed-by: Eiji Seki Discussion: https://postgr.es/m/20170228045643.n2ri74ara4fhhfxf@alvherre.pgsql http://git.postgresql.org/pg/commitdiff/c655899ba9ae2a0d24e99c797167c33e0cfa0820
  • BRIN auto-summarization. Previously, only VACUUM would cause a page range to get initially summarized by BRIN indexes, which for some use cases takes too much time since the inserts occur. To avoid the delay, have brininsert request a summarization run for the previous range as soon as the first tuple is inserted into the first page of the next range. Autovacuum is in charge of processing these requests, after doing all the regular vacuuming/ analyzing work on tables. This doesn't impose any new tasks on autovacuum, because autovacuum was already in charge of doing summarizations. The only actual effect is to change the timing, i.e. that it occurs earlier. For this reason, we don't go any great lengths to record these requests very robustly; if they are lost because of a server crash or restart, they will happen at a later time anyway. Most of the new code here is in autovacuum, which can now be told about "work items" to process. This can be used for other things such as GIN pending list cleaning, perhaps visibility map bit setting, both of which are currently invoked during vacuum, but do not really depend on vacuum taking place. The requests are at the page range level, a granularity for which we did not have SQL-level access; we only had index-level summarization requests via brin_summarize_new_values(). It seems reasonable to add SQL-level access to range-level summarization too, so add a function brin_summarize_range() to do that. Authors: Ãlvaro Herrera, based on sketch from Simon Riggs. Reviewed-by: Thomas Munro. Discussion: https://postgr.es/m/20170301045823.vneqdqkmsd4as4ds@alvherre.pgsql http://git.postgresql.org/pg/commitdiff/7526e10224f0792201e99631567bbe44492bbde4

Simon Riggs pushed:

Andres Freund pushed:

Fujii Masao pushed:

  • Simplify the example of VACUUM in documentation. Previously a detailed activity report by VACUUM VERBOSE ANALYZE was described as an example of VACUUM in docs. But it had been obsolete for a long time. For example, commit feb4f44d296b88b7f0723f4a4f3945a371276e0b updated the content of that activity report in 2003, but we had forgotten to update the example. So basically we need to update the example. But since no one cared about the details of VACUUM output and complained about that mistake for such long time, per discussion on hackers, we decided to get rid of the detailed activity report from the example and simplify it. Back-patch to all supported versions. Reported by Masahiko Sawada, patch by me. Discussion: https://postgr.es/m/CAD21AoAGA2pB3p-CWmTkxBsbkZS1bcDGBLcYVcvcDxspG_XAfA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/ec19693014ed48fa1d8c7e0ce450795f752c4456

Magnus Hagander pushed:

Andrew Dunstan pushed:

Kevin Grittner pushed:

  • Add transition table support to plpgsql. Kevin Grittner and Thomas Munro Reviewed by Heikki Linnakangas, David Fetter, and Thomas Munro with valuable comments and suggestions from many others http://git.postgresql.org/pg/commitdiff/59702716324ab9c07b02fb005dcf14c7f48c4632
  • Add infrastructure to support EphemeralNamedRelation references. A QueryEnvironment concept is added, which allows new types of objects to be passed into queries from parsing on through execution. At this point, the only thing implemented is a collection of EphemeralNamedRelation objects -- relations which can be referenced by name in queries, but do not exist in the catalogs. The only type of ENR implemented is NamedTuplestore, but provision is made to add more types fairly easily. An ENR can carry its own TupleDesc or reference a relation in the catalogs by relid. Although these features can be used without SPI, convenience functions are added to SPI so that ENRs can easily be used by code run through SPI. The initial use of all this is going to be transition tables in AFTER triggers, but that will be added to each PL as a separate commit. An incidental effect of this patch is to produce a more informative error message if an attempt is made to modify the contents of a CTE from a referencing DML statement. No tests previously covered that possibility, so one is added. Kevin Grittner and Thomas Munro Reviewed by Heikki Linnakangas, David Fetter, and Thomas Munro with valuable comments and suggestions from many others http://git.postgresql.org/pg/commitdiff/18ce3a4ab22d2984f8540ab480979c851dae5338
  • Try to fix breakage of sepgsql hooks by ENR patch. Turned up by buildfarm animal rhinoceros. Fixing blind. Will have to wait for next run by rhinoceros to know whether it worked. http://git.postgresql.org/pg/commitdiff/01fd6f8f2d15a9369768921d6fc95ac481779430
  • Fix two undocumented parameters to functions from ENR patch. On ProcessUtility document the parameter, to match others. On CreateCachedPlan drop the queryEnv parameter. It was not referenced within the function, and had been added on the assumption that with some unknown future usage of QueryEnvironment it might be useful to do something there. We have avoided other "just in case" implementation of unused paramters, so drop it here. Per gripe from Tom Lane http://git.postgresql.org/pg/commitdiff/41bd155dd656e7f17c02855be7aff234843347cd

Correctifs en attente

Kyotaro HORIGUCHI sent in a patch to add encoding_dumper and map_dumper.

Michaël Paquier sent in two more revisions of a patch to change detection of corrupted 2PC files as FATAL and minimize window between history file and end-of-recovery.

Nikhil Sontakke sent in another revision of a patch to speed up two-phase transactions.

Stas Kelvich and Craig Ringer traded patches to implement logical decoding of two-phase transactions.

Haribabu Kommi sent in three more revisions of a patch to implement a pg_stat_wal_write statistics view.

Ashutosh Bapat sent in two more revisions of a patch to implement partition-wise join for declaratively partitioned tables.

Kyotaro HORIGUCHI sent in a patch to fix a bug in physical replication slots.

Beena Emerson sent in another revision of a patch to allow increasing the default WAL segment size.

Amit Khandekar sent in two more revisions of a patch to enable UPDATE of partition key for declaratively partitioned tables.

Vaishnavi Prabakaran sent in two more revisions of a patch to implement batch/pipelining support for libpq.

Pavan Deolasee sent in six more revisions of a patch to implement WARM.

Mithun Cy sent in eight more revisions of a patch to expand hash indexes more efficiently.

Ashutosh Sharma sent in two more revisions of a patch to fix a failed assertion found by sqlsmith in _hash_kill_items/MarkBufferDirtyHint.

Pavel Stěhule and Surafel Temesgen traded patches to add CORRESPONDING.

Dmitry Dolgov sent in three more revisions of a patch to implement generic type subscripting.

Ashutosh Sharma sent in three more revisions of a patch to rewrite hash index scans to work a page at a time, remove redundant function _hash_step() and some of the unused members of HashScanOpaqueData, and improve the locking strategy during VACUUM in hash index.

Jan Michálek sent in four more revisions of a patch to add markdown and rst to psql's output formats.

Pavel Stěhule sent in a patch to add an xmltable doc fix and example for XMLNAMESPACES.

Pavel Stěhule and Petr Jelínek traded patches to add a plan_cache control PRAGMA to pl/pgsql.

Jeff Janes sent in a patch to fix an infelicity between free space map and visibility map.

Claudio Freire sent in another revision of a patch for VACUUM which frees the dead tuples array as early as possible.

Jim Nasby sent in a patch to fix a missing increment of vacrelstats->pinskipped_pages.

Ashutosh Bapat and Amit Langote traded patches to fix a comment in src/backend/optimizer/path/allpaths.c.

Haribabu Kommi sent in another revision of a patch to refactor handling of database attributes between pg_dump and pg_dumpall.

Kyotaro HORIGUCHI sent in a patch to show vacuum aggressiveness in autovacuum logs.

Alexander Law sent in a patch to remove trailing spaces in strings in the source code.

Daniel Gustafsson sent in a patch to use American English spelling in pg_waldump error message.

Nikhil Sontakke sent in another revision of a patch to move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM.

Alexander Korotkov sent in another revision of a patch to implement incremental sort.

Masahiko Sawada sent in another revision of a patch to implement support for transactions involving multiple postgres foreign servers.

Daniel Gustafsson sent in a patch to error out more informatively when multiple TO VERSION arguments are supplied to ALTER EXTENSION.

Takeshi Ideriha sent in another revision of a patch to implement DECLARE STATEMENT setting up a connection in ECPG.

Peter Eisentraut sent in a patch to adjust min/max values when changing sequence type.

Peter Eisentraut sent in another revision of a patch to implement IDENTITY columns.

Michaël Paquier sent in three more revisions of a patch to allow interrupts on waiting standby.

Vinayak Pokale sent in another revision of a patch to implement an ANALYZE command progress checker.

Tomas Vondra sent in two more revisions of a patch to add page_checksum and bt_page_items(bytea) to page_inspect.

Ashutosh Sharma sent in a patch to fix an issue that manifested as an inconsistent page found on STANDBY server.

Takayuki Tsunakawa sent in another revision of a patch to implement huge pages on Windows.

Kuntal Ghosh sent in a patch to fix some strange parallel query behavior after OOM crashes.

David Rowley sent in two more revisions of a patch to fix functional dependencies in materialized views in light of the extended statistics recently committed.

Thomas Munro sent in another revision of a patch to implement [[Parallel] Shared] Hash.

Anastasia Lubennikova and Teodor Sigaev traded patches to add covering + unique indexes.

Michaël Paquier sent in another revision of a patch to implement SASLprep aka NFKC for SCRAM authentication.

Aleksander Alekseev sent in a patch to remove an unused argument in btree_xlog_split.

Etsuro Fujita sent in another revision of a patch to add epqpath support for foreign joins.

Masahiko Sawada sent in a patch to remove some dead code from the table sync worker.

Fabien COELHO sent in a patch to refactor psql's ef/ev and sf/sv handling functions so they're not copypasta.

Feike Steenbergen sent in a patch to fix an issue where passwords in user mappings were leaked by psql's \deu+ command.

Alexander Korotkov sent in another revision of a patch to implement a LWLock optimization for multicore Power machines.

Takayuki Tsunakawa sent in a patch to fix an issue where savepoint-related statements in a multi-command query terminates the connection unexpectedly.

Mike Palmiotto sent in two revisions of a patch to silence some sepgsql compiler warnings and add partitioned table support to sepgsql.

Petr Jelínek sent in a patch to use weaker locks when updating subscription relation state.

Dilip Kumar sent in a patch to ensure that parallel bitmapscan is exercised in regression tests.

David Rowley sent in another revision of a patch to improve performance for outer joins where the outer side is unique.

par chl le mardi 4 avril 2017 à 22h08

mardi 21 mars 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 19 mars 2017

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170320021208.GB30492@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Remove dead code in nodeGatherMerge.c. Coverity noted that the last line of gather_merge_getnext() was unreachable, since each arm of the preceding "if" ends in a "return". Drop it as an oversight. In passing, improve some nearby comments. http://git.postgresql.org/pg/commitdiff/5d3f7c57ab9c9e2f074ad29d619056570fc5c51e
  • Fix typo in initdb's SCRAM password processing. Noted by Coverity (a rather impressive catch). Michael Paquier http://git.postgresql.org/pg/commitdiff/835cc1136745e8cf02d3d0231b5b7c7a543df5df
  • Add "break"s to make it clearer what will happen in a nested switch. This could only matter if the guessed_type variable had a value that wasn't a member of the PasswordType enum; but just in case, let's be sure that control falls out to reach the elog(ERROR) at the end of the function. Per gripe from Coverity. http://git.postgresql.org/pg/commitdiff/766f7fd613adbceaf1b40803793e10dc487f8596
  • Remove unnecessary dependency on statement_timeout in prepared_xacts test. Rather than waiting around for statement_timeout to expire, we can just try to take the table's lock in nowait mode. This saves some fraction under 4 seconds when running this test with prepared xacts available, and it guards against timeout-expired-anyway failures on very slow machines when prepared xacts are not available, as seen in a recent failure on axolotl for instance. This approach could fail if autovacuum were to take an exclusive lock on the test table concurrently, but there's no reason for it to do so. Since the main point here is to improve stability in the buildfarm, back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/1c7a66a8e9378aeb092d7ed26890134d17fdd691
  • Add a "void *" passthrough pointer for psqlscan.l's callback functions. The immediate motivation for this is to provide clean infrastructure for the proposed \if...\endif patch for psql; but it seems like a good thing to have even if that patch doesn't get in. Previously the callback functions could only make use of application-global state, which is a pretty severe handicap. For the moment, the pointer is only passed through to the get_variable callback function. I considered also passing it to the write_error callback, but for now let's not. Neither psql nor pgbench has a use for that, and in the case of psql we'd have to invent a separate wrapper function because we would certainly not want to change the signature of psql_error(). Discussion: https://postgr.es/m/10108.1489418309@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/895e36bb3f36fdb7ec8e573be1a20d104fac820b
  • Fix busted markup. Oversight in commit 9ca5c8721. Per buildfarm. http://git.postgresql.org/pg/commitdiff/0c87cd003d9966fcb19d6998ccf90d3276b08e0c
  • Make logging about multixact wraparound protection less chatty. The original messaging design, introduced in commit 068cfadf9, seems too chatty now that some time has elapsed since the bug fix; most installations will be in good shape and don't really need a reminder about this on every postmaster start. Hence, arrange to suppress the "wraparound protections are now enabled" message during startup (specifically, during the TrimMultiXact() call). The message will still appear if protection becomes effective at some later point. Discussion: https://postgr.es/m/17211.1489189214@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/5ed6fff6b729c3cce55d4abc8f695da93aa40a0d
  • Include port number when logging successful binding to a TCP port. Per suggestion from Andres Freund. Discussion: https://postgr.es/m/20170314033842.st7gifec55yigz2h@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/2b32ac2a59df18246c3b79e96a209bfdb39bb918
  • Rewrite async-connection loop in libpqwalreceiver.c, once again. The original coding in commit 1e8a85009 didn't use PQconnectPoll per spec, and while the rewrite in e434ad39a is closer, it still doesn't guarantee to wait until the socket is read-ready or write-ready (as appropriate) before calling PQconnectPoll. It's not clear whether that omission is causing the continuing failures on buildfarm member bowerbird; but given the lack of other explanations meeting the available facts, let's tighten that up and see what happens. An independent issue in the same loop was that it had a race condition whereby it could clear the process's latch without having serviced an interrupt request, causing failure to respond to a cancel while waiting for connection (the very problem 1e8a85009 was meant to fix). Discussion: https://postgr.es/m/7295.1489596949@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/b5dd50f2c0ad8edcc3145aabe18788d448bf940d
  • Fix WaitEventSetWait() to handle write-ready waits properly on Windows. Windows apparently will not detect socket write-ready events unless a preceding send attempt returned WSAEWOULDBLOCK. In many usage patterns that's satisfied by the caller of WaitEvenSetWait(), but not always. Apply the same solution that we already had in pgwin32_select(), namely to perform a dummy WSASend() call with len=0. This will return WSAEWOULDBLOCK if there's no buffer space (even though it could legitimately do nothing and report success, which makes me a bit nervous about this solution; but since it's been working fine in libpq, let's roll with it). In passing, improve the comments about this in pgwin32_select(), and remove duplicated code there. Back-patch to 9.6 where WaitEventSetWait() was introduced. We might need to back-patch something similar into predecessor code. But given the lack of complaints so far, it's not clear that the case ever gets exercised in the back branches, so I'm not going to expend effort on it right now. This should resolve recurring failures on buildfarm member bowerbird, which has been failing since 1e8a85009 went in. Diagnosis and patch by Petr Jelinek, cosmetic adjustments by me. Discussion: https://postgr.es/m/5b6a6d6d-fb45-0afb-2e95-5600063c3dbd@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/f7819baa618c528f60e266874051563ecfe08207
  • Fix REFRESH MATERIALIZED VIEW to report activity to the stats collector. The non-concurrent code path for REFRESH MATERIALIZED VIEW failed to report its updates to the stats collector. This is bad since it means auto-analyze doesn't know there's any work to be done. Adjust it to report the refresh as a table truncate followed by insertion of an appropriate number of rows. Since a matview could contain more than INT_MAX rows, change the signature of pgstat_count_heap_insert() to accept an int64 rowcount. (The accumulator it's adding into is already int64, but existing callers could not insert more than a small number of rows at once, so the argument had been declared just "int n".) This is surely a bug fix, but changing pgstat_count_heap_insert()'s API seems too risky for the back branches. Given the lack of previous complaints, I'm not sure it's a big enough problem to justify a kluge solution that would avoid that. So, no back-patch, at least for now. Jim Mlodgenski, adjusted a bit by me Discussion: https://postgr.es/m/CAB_5SRchSz7-WmdO5szdiknG8Oj_GGqJytrk1KRd11yhcMs1KQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/17f8ffa1e331cd0d95a3c4ccec66ea83d8b893c5
  • Avoid use of already-closed relcache entry. Oversight in commit 17f8ffa1e. Per buildfarm member prion. http://git.postgresql.org/pg/commitdiff/e3044f6184beac395e88b4e1230e6c9d449db7f2

Noah Misch pushed:

  • Assume deconstruct_array() outputs are untoasted. In functions that issue a deconstruct_array() call, consistently use plain VARSIZE()/VARDATA() on the array elements. Prior practice was divided between those and VARSIZE_ANY_EXHDR()/VARDATA_ANY(). http://git.postgresql.org/pg/commitdiff/2fd26b23b662dcb0cf649e983a58581cb911fc4b
  • Fix comment about length of text, bytea, etc. When commit 3e23b68dac006e8deb0afa327e855258df8de064 introduced single-byte varlena headers, it rendered this comment incomplete. http://git.postgresql.org/pg/commitdiff/9e0926468a1c41a31c09785787a737311dcd92c1
  • Recommend wrappers of PG_DETOAST_DATUM_PACKED(). When commit 3e23b68dac006e8deb0afa327e855258df8de064 introduced single-byte varlena headers, its fmgr.h changes presented PG_GETARG_TEXT_PP() and PG_GETARG_TEXT_P() as equals. Its postgres.h changes presented PG_DETOAST_DATUM_PACKED() and VARDATA_ANY() as the exceptional case. Now, instead, firmly recommend PG_GETARG_TEXT_PP() over PG_GETARG_TEXT_P(); likewise for other ...PP() macros. This shaves cycles and invites consistency of style. http://git.postgresql.org/pg/commitdiff/9d7726c2ba06b932f791f2d0cc5acf73cc0b4dca
  • Use wrappers of PG_DETOAST_DATUM_PACKED() more. This makes almost all core code follow the policy introduced in the previous commit. Specific decisions: * Text search support functions with char* and length arguments, such as prsstart and lexize, may receive unaligned strings. I doubt maintainers of non-core text search code will notice. * Use plain VARDATA() on values detoasted or synthesized earlier in the same function. Use VARDATA_ANY() on varlenas sourced outside the function, even if they happen to always have four-byte headers. As an exception, retain the universal practice of using VARDATA() on return values of SendFunctionCall(). * Retain PG_GETARG_BYTEA_P() in pageinspect. (Page images are too large for a one-byte header, so this misses no optimization.) Sites that do not call get_page_from_raw() typically need the four-byte alignment. * For now, do not change btree_gist. Its use of four-byte headers in memory is partly entangled with storage of 4-byte headers inside GBT_VARKEY, on disk. * For now, do not change gtrgm_consistent() or gtrgm_distance(). They incorporate the varlena header into a cache, and there are multiple credible implementation strategies to consider. http://git.postgresql.org/pg/commitdiff/3a0d473192b2045cbaf997df8437e7762d34f3ba
  • Fix pg_file_write() error handling. Detect fclose() failures; given "ln -s /dev/full $PGDATA/devfull", "pg_file_write('devfull', 'x', true)" now fails as it should. Don't leak a stream when fwrite() fails. Remove a born-ineffective test that aimed to skip zero-length writes. Back-patch to 9.2 (all supported versions). http://git.postgresql.org/pg/commitdiff/944a026b4ec252667f275768ba4dcd53ae3bb07e

Magnus Hagander pushed:

Peter Eisentraut pushed:

Heikki Linnakangas pushed:

Michael Meskes pushed:

Ãlvaro Herrera pushed:

Robert Haas pushed:

Andres Freund pushed:

Stephen Frost pushed:

  • Add support for EUI-64 MAC addresses as macaddr8. This adds in support for EUI-64 MAC addresses by adding a new data type called 'macaddr8' (using our usual convention of indicating the number of bytes stored). This was largely a copy-and-paste from the macaddr data type, with appropriate adjustments for having 8 bytes instead of 6 and adding support for converting a provided EUI-48 (6 byte format) to the EUI-64 format. Conversion from EUI-48 to EUI-64 inserts FFFE as the 4th and 5th bytes but does not perform the IPv6 modified EUI-64 action of flipping the 7th bit, but we add a function to perform that specific action for the user as it may be commonly done by users who wish to calculate their IPv6 address based on their network prefix and 48-bit MAC address. Author: Haribabu Kommi, with a good bit of rework of macaddr8_in by me. Reviewed by: Vitaly Burovoy, Kuntal Ghosh Discussion: https://postgr.es/m/CAJrrPGcUi8ZH+KkK+=TctNQ+EfkeCEHtMU_yo1mvX8hsk_ghNQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/c7a9fa399d557c6366222e90b35db31e45d25678
  • Bump catversion for MACADDR8. Pointed out by Robert. http://git.postgresql.org/pg/commitdiff/c5832346625af4193b1242e57e7d13e66a220b38
  • Clean up overly paranoid checks in mac8.c. Andres' compiler points out, quite correctly, that there's no need for some of the overly paranoid checks which were put into mac8.c. Remove those, as they're useless, add some comments and make a few other minor improvements- reduce the size of hexlookup by making it a char array instead of an int array, and pass in the ptr location directly instead of making hex2_to_uchar re-calculate the location based off the offset every time. http://git.postgresql.org/pg/commitdiff/7821f7229c6e149046ee0dd8cab57928c4f86a37
  • Be more careful about signed vs. unsigned char. The buildfarm has reminded me that not all systems consider char to be signed and we need to be explicit. Adjust the various bits of mac8.c for what we intend, mostly using casts to unsigned char as suggested by Tom, and adjust the tests for valid input accordingly. Explicitly make the hexlookup table signed as it's useful to use -1 there to indicate an invalid value. http://git.postgresql.org/pg/commitdiff/cccbddeb1483b85f1853a29dc3b6464647b91eee
  • Improve pg_dump regression tests and code coverage. These improvements bring the lines-of-code coverage of pg_dump.c up to 87.7% (at least using LCOV 1.12, 1.11 seems to differ slightly). Nearly every function is covered, three of the four which aren't are only called when talking to older PG instances. There is more which can, and should, be done here to improve the coverage but it's past time to see what the buildfarm thinks of this. What has been added: * Coverage for many more command-line options * Use command_fails_like instead of command_exit_is * Operator classes, operator families * Text search configuration, templates, parsers, dictionaries * FDWs, servers, foreign tables * Materialized views * Improved Publications / Subscriptions test (though this needs work, see PG10 open items and tests marked with XXX in 002_pg_dump.pl) * Unlogged tables * Partitioned tables * Additional ACL testing for various object types There is room for improvement, specifically: * Various type-based option (alignment, storage, etc) * Composite type collation * Extra Procedural language functions (inline, validator) * Different function options (SRF, Transform, config, security definer, cost, leakproof) * OpClass options (default, storage, order by, recheck) * OpFamily options (order by, recheck) * Aggregate functions (combinefunc, serialfunc, deserialfunc, etc) * Text Search parser 'headline' * Text Search template 'init' * FDW options (handler, validator, options) * Server options (type, version, options) * User mapping options * Default ACLs for sequences, types * Security labels * View circular dependencies (last function that needs coverage) * Toast table autovacuum options * Replica identity options * Independent indexes (plus marking them as clustered on) * Deferrable / initially deferred constraints * Independent domain constraints There's bits of extension pg_dump'ing also not covered, but those will need to go into test_pg_dump (such as having a filter for config tables). Last, but not least, this approximately halves the number of tests run with 'ok()' by removing the ok()-based checking of if all runs are covered by each test. Instead, 002_pg_dump.pl will just exit out in such a case (with a message in the log file). In general, when adding tests, cover all runs unless there is a very good reason not to (such as adding a 'catch-all' case). With these changes, the resulting output and number of "tests" run is actually reduced. http://git.postgresql.org/pg/commitdiff/31a8b77a9244a0883e1968adcff9b6038e575d77
  • pg_dump: Remove "option requires an argument -- j" test. This is really testing getopt more than pg_dump, and what getopt returns exactly appears to differ based on platform, so remove this test. Per buildfarm. http://git.postgresql.org/pg/commitdiff/5abda5a3e9e7f2a472ccbb1b8389d9166d4a9eca
  • Adjust number of tests for pg_dump 001_basic.pl. When removing a test, need to make sure the count of tests is adjusted when it isn't calculated. http://git.postgresql.org/pg/commitdiff/805e8bc7029b6eb19f0ca3a68051cfda5bd07ef3
  • pg_dump: Skip COLLATION-related regression tests. Not every platform supports non-default collations, as pointed out by the buildfarm, so skip collation-related regression tests in pg_dump when they aren't supported. http://git.postgresql.org/pg/commitdiff/de34123834c3fa465b97f65ded171888edbfbccf

Andrew Gierth pushed:

  • Avoid having vacuum set reltuples to 0 on non-empty relations in the presence of page pins, which leads to serious estimation errors in the planner. This particularly affects small heavily-accessed tables, especially where locking (e.g. from FK constraints) forces frequent vacuums for mxid cleanup. Fix by keeping separate track of pages whose live tuples were actually counted vs. pages that were only scanned for freezing purposes. Thus, reltuples can only be set to 0 if all pages of the relation were actually counted. Backpatch to all supported versions. Per bug #14057 from Nicolas Baccelli, analyzed by me. Discussion: https://postgr.es/m/20160331103739.8956.94469@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/1914c5ea7daaaaba4420f65c991256af5d4a9813
  • Repair test for vacuum reltuples fix. Concurrent auto-analyze could be holding a snapshot, affecting the removal of deleted row versions. Remove the deletion to avoid this happening. Per buildfarm. In passing, make the test independent of assumptions of physical row order, just out of sheer paranoia. http://git.postgresql.org/pg/commitdiff/64ae420b275a82534732aafd9d550b9982ca0a5d

Correctifs en attente

Andreas Karlsson sent in another revision of a patch to implement REINDEX CONCURRENTLY.

Amit Langote sent in a patch to fix a bug in get_partition_for_tuple.

KaiGai Kohei sent in another revision of a patch to add PassDownLimitBound for ForeignScan/CustomScan.

Masahiko Sawada sent in another revision of a patch to support 2PC for FDWs.

Pavel Stěhule sent in a patch to fix an xpath bug.

Craig Ringer and Petr Jelínek traded patches to add an option to control snapshot export to CREATE_REPLICATION_SLOT and add logical replication support for initial data copy.

Craig Ringer sent in two more revisions of a patch to add a pg_recvlogical wrapper to PostgresNode, follow timeline switches in logical decoding, and enable logical decoding on standby.

Mithun Cy sent in another revision of a patch to implement auto_prewarm.

Thomas Munro sent in a patch to fix a bug in psql's tab completion for UPDATE.

Jeff Janes sent in a patch to silence a perl warning in the check-world make target.

Peter Geoghegan sent in a patch to avoid copying within tuplesort_gettupleslot().

Thomas Munro sent in a patch to update the "tool sets" documentation for modern FreeBSD.

David Rowley sent in two more revisions of a patch to improve performance for joins where outer side is unique.

Anastasia Lubennikova and Ashutosh Bapat traded patches to add IF NOT EXISTS option for CREATE SERVER and CREATE USER MAPPING statements.

Rafia Sabih sent in another revision of a patch to add parallelism support for queries coming from SQL or other PL functions.

Thomas Munro sent in another revision of a patch to help measure replay lag.

Vaishnavi Prabakaran sent in a patch to fix a bug in libpq's single row mode.

Michaël Paquier sent in another revision of a patch to change references of password encryption to hashing.

Amit Langote sent in four more revisions of a patch to avoid creating scan nodes for partitioned tables and not allocate storage for partitioned tables.

David Steele sent in two more revisions of a patch to make pg_stop_backup() archive wait optional.

Andres Freund, Heikki Linnakangas, and Andreas Karlsson traded patches to process expressions faster.

Aleksander Alekseev sent in another revision of a patch to suppress clang 3.9 warnings.

Beena Emerson sent in another revision of a patch to allow increasing the default WAL segment size.

Amit Kapila sent in another revision of a patch to enable parallel queries containing initplans.

Michaël Paquier sent in another revision of a patch to fix backend crash on non-exclusive backup cancel.

Ãlvaro Herrera and David Rowley traded patches to add extended (multivariate) statistics.

Ashutosh Sharma sent in six more revisions of a patch to add microvacuum support for Hash Index.

Surafel Temesgen and Pavel Stěhule traded patches to add CORRESPONDING.

Ashutosh Bapat and Rafia Sabih traded patches to support partition-wise join for join between (declaratively) partitioned tables.

Pavan Deolasee sent in two more revisions of a patch to implement WARM.

Masahiko Sawada sent in another revision of a patch to report the number of skipped frozen pages by manual VACUUM.

Kuntal Ghosh sent in a patch to show extParams and allParams in explain.

Andrew Gierth sent in another revision of a patch to add hash support for GROUPING SETS.

Haribabu Kommi sent in another revision of a patch to mark utility commands benefiting from parallel plan.

Haribabu Kommi sent in another revision of a patch to add pg_stat_wal_write statistics view.

Vaishnavi Prabakaran sent in another revision of a patch to add pipelining batch support for libpq code and tests for same.

Dilip Kumar sent in three revisions of a patch to fix a bug in parallel bitmap scans.

Nikhil Sontakke sent in another revision of a patch to speed up 2PC transactions.

Kuntal Ghosh sent in two more revisions of a patch to add infra to expose all backend processes in pg_stat_get_activity, expose stats for all backends, and add backend_type column in pg_stat_get_activity.

Heikki Linnakangas sent in a patch to allow SCRAM authentication, when pg_hba.conf says 'md5'.

Michaël Paquier sent in another revision of a patch to use base64-based encoding for stored and server keys in SCRAM verifiers, refactor frontend-side random number generation, move routine to build SCRAM verifier into src/common/, and extend PQencryptPassword with a hashing method.

Simon Riggs and David Rowley traded patches to improve the performance of replay of AccessExclusiveLock.

Peter Eisentraut sent in another revision of a patch to refine rules for altering publication owner, change logical replication pg_hba.conf use, add USAGE privilege for publications, add subscription apply worker privilege checks, and add CREATE SUBSCRIPTION privilege on databases.

Rushabh Lathia sent in two more revisions of a patch to add wait events for disk I/O.

Peter Eisentraut sent in another revision of a patch to implement ICU integration.

Thomas Munro sent in two revisions of a patch to change ancient use of Size to the now-common size_t.

Michaël Paquier sent in a patch to add clause PASSWORD (val USING protocol) to CREATE/ALTER ROLE.

Takayuki Tsunakawa sent in a patch to fix an issue where pgwin32_is_service was not checking whether SECURITY_SERVICE_SID was disabled.

Michael Banck sent in three revisions of a patch to reorder tablespaces for non-WAL streaming basebackups.

Amit Khandekar sent in another revision of a patch to implement parallel append.

Michaël Paquier sent in another revision of a patch to rename pg_subtrans to pg_subxact.

Corey Huinker sent in two more revisions of a patch to add \if and friends to psql.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous execution.

Dave Page sent in another revision of a patch to add monitoring roles.

Emre Hasegeli sent in another revision of a patch to add new BRIN cost estimates.

Yugo Nagata sent in another revision of a patch to implement hash partitioning.

Robert Haas sent in a patch to put make a single point of truth for adjust_relid_set while keeping it static.

Petr Jelínek sent in a patch to fix wait for writable socket on Windows.

Pritam Baral sent in a patch to enable using indexes for element-contained-by-const-range clauses.

Nikolay Shaplov sent in another revision of a patch to move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM.

Peter Eisentraut sent in a patch to pageinspect to add bt_page_items function with a bytea argument.

Elvis Pranskevichus sent in a patch to add and report the new "in_hot_standby" GUC pseudo-variable.

Vinayak Pokale sent in another revision of a patch to add an ANALYZE command progress checker.

Petr Jelínek sent in another revision of a patch to add an option to modify sync commit per subscription in logical replication.

Pavel Stěhule sent in another revision of a patch to psql to add \gload for loading binary formats.

Mithun Cy sent in another revision of a patch to expand hash buckets efficiently.

Jan Michálek sent in another revision of a patch to add markdown and rst output formats to psql.

Michael Banck sent in two revisions of a patch to create replication slot in pg_basebackup if requested and not yet present.

Peter Eisentraut sent in a patch to remove createlang and droplang.

Peter Geoghegan sent in another revision of a patch to add parallel B-tree index build sorting and add temporary testing tools.

par chl le mardi 21 mars 2017 à 19h50

mardi 14 mars 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 12 mars 2017

Le PostgresOpen aura lieu à San Francisco du 6 au 8 septembre : https://2017.postgresopen.org/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170312234158.GB31409@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Simon Riggs pushed:

Peter Eisentraut pushed:

Robert Haas pushed:

Tom Lane pushed:

  • Avoid dangling pointer to relation name in RLS code path in DoCopy(). With RLS active, "COPY tab TO ..." failed under -DRELCACHE_FORCE_RELEASE, and would sometimes fail without that, because it used the relation name directly from the relcache as part of the parsetree it's building. That becomes a potentially-dangling pointer as soon as the relcache entry is closed, a bit further down. Typical symptom if the relcache entry chanced to get cleared would be "relation does not exist" error with a garbage relation name, or possibly a core dump; but if you were really truly unlucky, the COPY might copy from the wrong table. Per report from Andrew Dunstan that regression tests fail with -DRELCACHE_FORCE_RELEASE. The core tests now pass for me (but have not tried "make check-world" yet). Discussion: https://postgr.es/m/7b52f900-0579-cda9-ae2e-de5da17090e6@2ndQuadrant.com http://git.postgresql.org/pg/commitdiff/a8df75b0a470f477dad75a7408e429e10c13fc07
  • Repair incorrect pg_dump labeling for some comments and security labels. We attached no schema label to comments for procedural languages, casts, transforms, operator classes, operator families, or text search objects. The first three categories of objects don't really have schemas, but pg_dump treats them as if they do, and it seems like the TocEntry fields for their comments had better match the TocEntry fields for the parent objects. (As an example of a possible hazard, the type names in a CAST will be formatted with the assumption of a particular search_path, so failing to ensure that this same path is active for the COMMENT ON command could lead to an error or to attaching the comment to the wrong cast.) In the last six cases, this was a flat-out error --- possibly mine to begin with, but it was a long time ago. The security label for a procedural language was likewise not correctly labeled as to schema, and both the comment and security label for a procedural language were not correctly labeled as to owner. In simple cases the restore would accidentally work correctly anyway, since these comments and security labels would normally get emitted right after the owning object, and so the search path and active user would be correct anyhow. But it could fail in corner cases; for example a schema-selective restore would omit comments it should include. Giuseppe Broccolo noted the oversight, and proposed the correct fix, for text search dictionary objects; I found the rest by cross-checking other dumpComment() calls. These oversights are ancient, so back-patch all the way. Discussion: https://postgr.es/m/CAFzmHiWwwzLjzwM4x5ki5s_PDMR6NrkipZkjNnO3B0xEpBgJaA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/299990ba16ab75b09b7fed61322f1f23fdc7cf4c
  • Remove vestigial grammar support for CHARACTER ... CHARACTER SET option. The SQL standard says that you should be able to write "CHARACTER SET foo" as part of the declaration of a char-type column. We don't implement that, but a rough form of support has existed in gram.y since commit f10b63923. That's now sat there for nigh 20 years without anyone fleshing it out --- and even if someone did, the contemplated approach of having separate data type name(s) for every character set certainly isn't what we'd do today. Let's just remove the grammar production; if anyone is ever motivated to work on this, reinventing the grammar support is a trivial fraction of what they'd have to do. And we've never documented anything about supporting such a clause. Per gripe from Neha Khatri. Discussion: https://postgr.es/m/CAFO0U+-iOS5oYN5v3SBuZvfhPUTRrkDFEx8w7H17B07Rwg3YUA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/11324e408f0e3a25621c611467927c644894b30d
  • Fix pgbench's failure to honor the documented long-form option "--builtin". Not only did it not accept --builtin as a synonym for -b, but what it did accept as a synonym was --tpc-b (huh?), which it got even further wrong by marking as no_argument, so that if you did try that you got a core dump. I suppose this is leftover from some early design for the new switches added by commit 8bea3d221, but it's still pretty sloppy work. Per bug #14580 from Stepan Pesternikov. Back-patch to 9.6 where the error was introduced. Report: https://postgr.es/m/20170307123347.25054.73207@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/ef2662394455578f6c57e99a7896c69bdd9fbd74
  • Clean up test_ifaddrs a bit. We customarily #include <netinet/in.h> before <arpa/inet.h>; according to our git history (cf commit 527f8babc) there used to be platform(s) where <arpa/inet.h> didn't compile otherwise. That's probably not really an issue anymore, but since test_ifaddrs.c is the one and only place in our code that's not following that rule, bring it into line. Also remove #include <sys/socket.h>, as that's duplicative given that libpq/ifaddr.h does so (via pqcomm.h). In passing, add a .gitignore file so nobody accidentally commits the test_ifaddrs executable, as I nearly did. I see no particular need to back-patch this, as it's just neatnik-ism considering we don't build test_ifaddrs by default, or even document it anywhere. http://git.postgresql.org/pg/commitdiff/03cf2219346aa78ecd1b6d4501a7697692a43c62
  • Invent start_proc parameters for PL/Tcl. Define GUCs pltcl.start_proc and pltclu.start_proc. When set to a nonempty value at the time a new Tcl interpreter is created, the parameterless pltcl or pltclu function named by the GUC is called to allow user-controlled initialization to occur within the interpreter. This is modeled on plv8's start_proc parameter, and also has much in common with plperl's on_init feature. It allows users to fully replace the "modules" feature that was removed in commit 817f2a586. Since an initializer function could subvert later Tcl code in nearly arbitrary ways, mark both GUCs as SUSET for now. It would be nice to find a way to relax that someday; but the corresponding GUCs in plperl are also SUSET, and there's not been much complaint. Discussion: https://postgr.es/m/22067.1488046447@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0d2b1f305dc78d536d80cfb4bb2ac4d7104453db
  • Silence compiler warnings in tbm_prepare_shared_iterate(). Maybe Robert's compiler can convince itself that these variables are never used uninitialized, but mine can't. http://git.postgresql.org/pg/commitdiff/270d7dd8a5a7128fc2b859f3bf95e2c1fb45be79
  • Use doubly-linked block lists in aset.c to reduce large-chunk overhead. Large chunks (those too large for any palloc freelist) are managed as separate blocks. Formerly, realloc'ing or pfree'ing such a chunk required O(N) time in a context with N blocks, since we had to traipse down the singly-linked block list to locate the block's predecessor before we could fix the list links. This can result in O(N^2) runtime in situations where large numbers of such chunks are manipulated within one context. Cases like that were not foreseen in the original design of aset.c, and indeed didn't arise until fairly recently. But such problems can now occur in reorderbuffer.c and in hash joining, both of which make repeated large requests without scaling up their request size as they do so, and which will free their requests in not-necessarily-LIFO order. To fix, change the block list from singly-linked to doubly-linked. This adds another 4 or 8 bytes to ALLOC_BLOCKHDRSZ, but that doesn't seem like unacceptable overhead, since aset.c's blocks are normally 8K or more, and never less than 1K in current practice. In passing, get rid of some redundant AllocChunkGetPointer() calls in AllocSetRealloc (the compiler might be smart enough to optimize these away anyway, but no need to assume that) and improve AllocSetCheck's checking of block header fields. Back-patch to 9.4 where reorderbuffer.c appeared. We could take this further back, but currently there's no evidence that it would be useful. Discussion: https://postgr.es/m/CAMkU=1x1hvue1XYrZoWk_omG0Ja5nBvTdvgrOeVkkeqs71CV8g@mail.gmail.com http://git.postgresql.org/pg/commitdiff/ff97741bc810390db6dd4da0f31ee1e93c8d3abb
  • Silence compiler warnings in BitmapHeapNext(). Same disease as 270d7dd8a5a7128fc2b859f3bf95e2c1fb45be79. http://git.postgresql.org/pg/commitdiff/15d03e597662847e13428a8b3ce9dd59c38decf3
  • Put back <float.h> in a few files that need it for _isnan(). Further fallout from commit c29aff959: there are some files that need <float.h>, and were getting it from datatype/timestamp.h, but it was not apparent in my (tgl's) testing because the requirement for <float.h> exists only on certain Windows toolchains. Report and patch by David Rowley. Discussion: https://postgr.es/m/CAKJS1f-BHceaFzZScFapDV48gUVM2CAOBfhkgffdqXzFb+kwew@mail.gmail.com http://git.postgresql.org/pg/commitdiff/86dbbf20d8496ede77566873d1e22cd11c1e544c
  • Suppress compiler warning in non-USE_LIBXML builds. Compilers that don't realize that ereport(ERROR) doesn't return complained that XmlTableGetValue() failed to return a value. Also, make XmlTableFetchRow's non-USE_LIBXML case look more like the other ones. As coded, it could lead to "unreachable code" warnings with USE_LIBXML enabled. Oversights in commit fcec6caaf. Per buildfarm. http://git.postgresql.org/pg/commitdiff/f379121093deb5465b01d93ebe9410d96c6cd093
  • Suppress compiler warning in slab.c. Compilers that don't realize that elog(ERROR) doesn't return complained that SlabRealloc() failed to return a value. While at it, fix the rather muddled header comment for the function. Per buildfarm. http://git.postgresql.org/pg/commitdiff/2f899e7d37ece937740c99164dd846c4b6f884eb
  • Document intentional violations of header inclusion policy. Although there are good reasons for our policy of including postgres.h as the first #include in every .c file, never from .h files, there are two places where it seems expedient to violate the policy because the alternative is to modify externally-supplied .c files. (In the case of the regexp library, the idea that it's externally-supplied is kind of at odds with reality, but I haven't entirely given up hope that it will become a standalone project some day.) Add some comments to make it explicit that this is a policy violation and provide the reasoning. In passing, move #include "miscadmin.h" out of regcomp.c and into regcustom.h, which is where it should be if we're taking this reasoning seriously at all. Discussion: https://postgr.es/m/CAEepm=2zCoeq3QxVwhS5DFeUh=yU6z81pbWMgfOB8OzyiBwxzw@mail.gmail.com Discussion: https://postgr.es/m/11634.1488932128@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/d6b059ec740a6affce9a069f1210d161068317e3
  • Bring plpgsql into line with header inclusion policy. We have a project policy that every .c file should start by including postgres.h, postgres_fe.h, or c.h as appropriate; and then there is no need for any .h file to explicitly include any of these. (The core reason for this policy is to make it easy to verify that pg_config_os.h is included before any system headers such as <stdio.h>; without that, we have portability issues on some platforms due to variation in largefile options across different modules in the backend. Also, if .h files were responsible for choosing which of these key headers to include, .h files that need to be includable in either frontend or backend compiles would be in trouble.) plpgsql was blithely ignoring this policy, so whack it upside the head until it complies. I also chose to standardize on including plpgsql's own .h files after all core-system headers that it pulls in. That could've been done either way, but this way seems saner. Discussion: https://postgr.es/m/CAEepm=2zCoeq3QxVwhS5DFeUh=yU6z81pbWMgfOB8OzyiBwxzw@mail.gmail.com Discussion: https://postgr.es/m/11634.1488932128@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/08da52859a1fadeac10aab621c6c793791ec1f2c
  • Fix inclusions of postgres_fe.h from .h files. We have a project policy that every .c file should start by including postgres.h, postgres_fe.h, or c.h as appropriate; and then there is no need for any .h file to explicitly include any of these. Fix a few headers that were violating this policy by including postgres_fe.h. Discussion: https://postgr.es/m/CAEepm=2zCoeq3QxVwhS5DFeUh=yU6z81pbWMgfOB8OzyiBwxzw@mail.gmail.com Discussion: https://postgr.es/m/11634.1488932128@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/9722bb5757c5e90617be685bf127911b63efe08d
  • Fix inclusions of c.h from .h files. We have a project policy that every .c file should start by including postgres.h, postgres_fe.h, or c.h as appropriate; and then there is no need for any .h file to explicitly include any of these. Fix a few headers that were violating this policy by including c.h. Discussion: https://postgr.es/m/CAEepm=2zCoeq3QxVwhS5DFeUh=yU6z81pbWMgfOB8OzyiBwxzw@mail.gmail.com Discussion: https://postgr.es/m/11634.1488932128@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/a72f0365db4168e7d720608fe3ac0ca3fe9355df
  • Fix timestamptz regression test to still work with latest IANA zone data. The IANA timezone crew continues to chip away at their project of removing timezone abbreviations that have no real-world currency from their database. The tzdata2017a update removes all such abbreviations for South American zones, as well as much of the Pacific. This breaks some test cases in timestamptz.sql that were expecting America/Santiago and America/Caracas to have non-numeric abbreviations. The test cases involving America/Santiago seem to have selected that zone more or less at random, so just replace it with America/New_York, which is of similar longitude. The cases involving America/Caracas are harder since they were chosen to test a time-varying zone abbreviation around a point where it changed meaning in the backwards direction. Fortunately, Europe/Moscow has a similar case in 2014, and the MSK/MSD abbreviations are well enough attested that IANA seems unlikely to decide to remove them from the database in future. With these changes, this regression test should pass when using any IANA zone database from 2015 or later. One could wish that there were a few years more daylight on how out-of-date your zone database can be ... but really the --with-system-tzdata option is only meant for use on platforms where the zone database is kept up-to-date pretty faithfully, so I do not think this is a big objection. Discussion: https://postgr.es/m/6749.1489087470@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/f077e1b2e374fadc443b6aa9fa54ad9bb6f908c7
  • Fix hard-coded relkind constants in pg_dump.c. Although it's reasonable to expect that most of these constants will never change, that does not make it good programming style to hard-code the value rather than using the RELKIND_FOO macros. There were only a few such violations, and all relatively new AFAICT. Existing style is mostly to inject relkind values into constructed query strings using %c. I did not bother to touch places that did it like that, but really a better technique is to stringify the RELKIND macro, at least in places where you'd want single quotes around the code character. That avoids any runtime effort and keeps the RELKIND symbol close to where it's used. Discussion: https://postgr.es/m/11145.1488931324@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/fe797b4a6a69ec0c9bf6ff992eac20c4084fda41
  • Make CppAsString2() more visible in c.h. For some reason this standard C string-processing hack was buried in an NLS-related section of c.h. Put it beside CppAsString() so that people are more likely to find it and not be tempted to reinvent local copies, as I nearly did. And provide a more helpful comment, too. http://git.postgresql.org/pg/commitdiff/9cfc4deeb9e6bd8162bd85dce5809d4d7aea2b66
  • Fix hard-coded relkind constants in psql/describe.c. Although it's reasonable to expect that most of these constants will never change, that does not make it good programming style to hard-code the value rather than using the RELKIND_FOO macros. Discussion: https://postgr.es/m/11145.1488931324@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/395bfaae8e786eb17fc9dd79e4636f97c9d9b820
  • Fix portability problem in Catalog.pm. Commit 7666e73a2 introduced a dependency on filehandles' input_line_number method, but apparently that's a Perl neologism. Use $. instead, which works at least back to Perl 5.10, and hopefully back to 5.8. Jeff Janes Discussion: https://postgr.es/m/CAMkU=1wuQW=xVfu-14A4VCvxO0ohkD3m9vk6HOj_dprQoKNAQw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/15bb93e28e49fdf4f28d509c07d1527886acb3e2
  • Fix hard-coded relkind constants in assorted src/bin files. Although it's reasonable to expect that most of these constants will never change, that does not make it good programming style to hard-code the value rather than using the RELKIND_FOO macros. Discussion: https://postgr.es/m/11145.1488931324@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/fcd778eb703c154c0fbba249e17c21b7ae4de19b
  • Add .gitignore to contrib/amcheck. Oversight in commit 3717dc149. http://git.postgresql.org/pg/commitdiff/574268e37b3b7b3449eb0034013443d88f4bd320
  • Fix hard-coded relkind constants in assorted other files. Although it's reasonable to expect that most of these constants will never change, that does not make it good programming style to hard-code the value rather than using the RELKIND_FOO macros. I think I've now gotten all the hard-coded references in C code. Unfortunately there's no equally convenient way to parameterize SQL files ... Discussion: https://postgr.es/m/11145.1488931324@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/9c2635e26f6f4e34b3b606c0fc79d0e111953a74
  • Un-break things on IPv6-less platforms. Commit be37c2120 forgot to teach initdb about commenting out the IPv6 replication entry that it caused to exist in pg_hba.conf.sample. Per buildfarm. http://git.postgresql.org/pg/commitdiff/a83e4b4f31c7afa5f7360086ebb1916cc99a4dbe
  • Change the relkind for partitioned tables from 'P' to 'p'. Seven of the eight other relkind codes are lower-case, so it wasn't consistent for this one to be upper-case. Fix it while we still can. Historical notes: the reason for the lone exception, i.e. sequences being 'S', is that 's' was once used for "special" relations. Also, at one time the partitioned-tables patch used both 'P' and 'p', but that got changed, leaving only a surprising choice behind. This also fixes a couple little bits of technical debt, such as type_sanity.sql not knowing that 'm' is a legal value for relkind. Discussion: https://postgr.es/m/27899.1488909319@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/8b358b42f8eb6156a82ac9a41fc4e8335c8dc37a
  • Sanitize newlines in object names in "pg_restore -l" output. Commits 89e0bac86 et al replaced newlines with spaces in object names printed in SQL comments, but we neglected to consider that the same names are also printed by "pg_restore -l", and a newline would render the output unparseable by "pg_restore -L". Apply the same replacement in "-l" output. Since "pg_restore -L" doesn't actually examine any object names, only the dump ID field that starts each line, this is enough to fix things for its purposes. The previous fix was treated as a security issue, and we might have done that here as well, except that the issue was reported publicly to start with. Anyway it's hard to see how this could be exploited for SQL injection; "pg_restore -L" doesn't do much with the file except parse it for leading integers. Per bug #14587 from Milos Urbanek. Back-patch to all supported versions. Discussion: https://postgr.es/m/20170310155318.1425.30483@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/f39ddd843667c68f760cb4cf23c89c1f1d9aebb8
  • Reduce log verbosity of startup/shutdown for launcher subprocesses. There's no really good reason why the autovacuum launcher and logical replication launcher should announce themselves at startup and shutdown by default. Users don't care that those processes exist, and it's inconsistent that those background processes announce themselves while others don't. So, reduce those messages from LOG to DEBUG1 level. I was sorely tempted to reduce the "starting logical replication worker for subscription ..." message to DEBUG1 as well, but forebore for now. Those processes might possibly be of direct interest to users, at least until logical replication is a lot better shaken out than it is today. Discussion: https://postgr.es/m/19479.1489121003@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/6ec4c8584c45ee784a95e02c40bb643430dee32a
  • contrib/amcheck needs RecentGlobalXmin to be PGDLLIMPORT'ified. Per buildfarm. Maybe some of the other xmin variables in snapmgr.h ought to get this too, but for the moment I'm just interested in un-breaking the buildfarm. http://git.postgresql.org/pg/commitdiff/56018bf26eec1a0b4bf20303c98065a8eb1b0c5d
  • Improve postmaster's logging of listen socket creation. When one of the kernel calls in the socket()/bind()/listen() sequence fails, include the specific address we're trying to bind to in the log message. This greatly eases debugging of network misconfigurations. Also, after successfully setting up a listen socket, report its address in the log, to ease verification that the expected addresses were bound. There was some debate about whether to print this message at LOG level or only DEBUG1, but the majority of votes were for the former. Discussion: https://postgr.es/m/9564.1489091245@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/f9dfa5c9776649f769d537dd0923003b35f128de
  • Add a "subtransaction" command to PL/Tcl. This allows rolling back the effects of some SPI commands without having to fail the entire PL/Tcl function. Victor Wagner, reviewed by Pavel Stehule Discussion: https://postgr.es/m/20170108205750.2dab04a1@wagner.wagner.home http://git.postgresql.org/pg/commitdiff/b58fd4a9cab21e9d937a4e369bab31b3a5d24710

Stephen Frost pushed:

  • pg_upgrade: Fix large object COMMENTS, SECURITY LABELS. When performing a pg_upgrade, we copy the files behind pg_largeobject and pg_largeobject_metadata, allowing us to avoid having to dump out and reload the actual data for large objects and their ACLs. Unfortunately, that isn't all of the information which can be associated with large objects. Currently, we also support COMMENTs and SECURITY LABELs with large objects and these were being silently dropped during a pg_upgrade as pg_dump would skip everything having to do with a large object and pg_upgrade only copied the tables mentioned to the new cluster. As the file copies happen after the catalog dump and reload, we can't simply include the COMMENTs and SECURITY LABELs in pg_dump's binary-mode output but we also have to include the actual large object definition as well. With the definition, comments, and security labels in the pg_dump output and the file copies performed by pg_upgrade, all of the data and metadata associated with large objects is able to be successfully pulled forward across a pg_upgrade. In 9.6 and master, we can simply adjust the dump bitmask to indicate which components we don't want. In 9.5 and earlier, we have to put explciit checks in in dumpBlob() and dumpBlobs() to not include the ACL or the data when in binary-upgrade mode. Adjustments made to the privileges regression test to allow another test (large_object.sql) to be added which explicitly leaves a large object with a comment in place to provide coverage of that case with pg_upgrade. Back-patch to all supported branches. Discussion: https://postgr.es/m/20170221162655.GE9812@tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/ff992c074e308ade204a38eb43a6d19c8403414e
  • pg_dump: Properly handle public schema ACLs with --clean. pg_dump has always handled the public schema in a special way when it comes to the "--clean" option. To wit, we do not drop or recreate the public schema in "normal" mode, but when we are run in "--clean" mode then we do drop and recreate the public schema. When running in "--clean" mode, the public schema is dropped and then recreated and it is recreated with the normal schema-default privileges of "nothing". This is unlike how the public schema starts life, which is to have CREATE and USAGE GRANT'd to the PUBLIC role, and that is what is recorded in pg_init_privs. Due to this, in "--clean" mode, pg_dump would mistakenly only dump out the set of privileges required to go from the initdb-time privileges on the public schema to whatever the current-state privileges are. If the privileges were not changed from initdb time, then no privileges would be dumped out for the public schema, but with the schema being dropped and recreated, the result was that the public schema would have no ACLs on it instead of what it should have, which is the initdb-time privileges. Practically speaking, this meant that pg_dump with --clean mode dumping a database where the ACLs on the public schema were not changed from the default would, upon restore, result in a public schema with *no* privileges GRANT'd, not matching the state of the existing database (where the initdb-time privileges would have been CREATE and USAGE to the PUBLIC role for the public schema). To fix, adjust the query in getNamespaces() to ignore the pg_init_privs entry for the public schema when running in "--clean" mode, meaning that the privileges for the public schema would be dumped, correctly, as if it was going from a newly-created schema to the current state (which is, indeed, what will happen during the restore thanks to the DROP/CREATE). Only the public schema is handled in this special way by pg_dump, no other initdb-time objects are dropped/recreated in --clean mode. Back-patch to 9.6 where the bug was introduced. Discussion: https://postgr.es/m/3534542.o3cNaKiDID%40techfox http://git.postgresql.org/pg/commitdiff/330b84d8c40864007833e05dc9d849c4bda77240
  • psql: Add \gx command. It can often be useful to use expanded mode output (\x) for just a single query. Introduce a \gx which acts exactly like \g except that it will force expanded output mode for that one \gx call. This is simpler than having to use \x as a toggle and also means that the user doesn't have to worry about the current state of the expanded variable, or resetting it later, to ensure a given query is always returned in expanded mode. Primairly Christoph's patch, though I did tweak the documentation and help text a bit, and re-indented the tab completion section. Author: Christoph Berg Reviewed By: Daniel Verite Discussion: https://postgr.es/m/20170127132737.6skslelaf4txs6iw%40msg.credativ.de http://git.postgresql.org/pg/commitdiff/b2678efd43f17db7dfa04e0ca076ea01275cd9bc
  • Expose explain's SUMMARY option. This exposes the existing explain summary option to users to allow them to choose if they wish to have the planning time and totalled run time included in the EXPLAIN result. The existing default behavior is retained if SUMMARY is not specified- running explain without analyze will not print the summary lines (just the planning time, currently) while running explain with analyze will include the summary lines (both the planning time and the totalled execution time). Users who wish to see the summary information for plain explain can now use: EXPLAIN (SUMMARY ON) query; Users who do not want to have the summary printed for an analyze run can use: EXPLAIN (ANALYZE ON, SUMMARY OFF) query; With this, we can now also have EXPLAIN ANALYZE queries included in our regression tests by using: EXPLAIN (ANALYZE ON, TIMING OFF, SUMMARY off) query; I went ahead and added an example of this, which will hopefully not make the buildfarm complain. Author: Ashutosh Bapat Discussion: https://postgr.es/m/CAFjFpReE5z2h98U2Vuia8hcEkpRRwrauRjHmyE44hNv8-xk+XA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/f9b1a0dd403ec0931213c66d5f979a3d3e8e7e30
  • Add relkind checks to certain contrib modules. The contrib extensions pageinspect, pg_visibility and pgstattuple only work against regular relations which have storage. They don't work against foreign tables, partitioned (parent) tables, views, et al. Add checks to the user-callable functions to return a useful error message to the user if they mistakenly pass an invalid relation to a function which doesn't accept that kind of relation. In passing, improve some of the existing checks to use ereport() instead of elog(), add a function to consolidate common checks where appropriate, and add some regression tests. Author: Amit Langote, with various changes by me Reviewed by: Michael Paquier and Corey Huinker Discussion: https://postgr.es/m/ab91fd9d-4751-ee77-c87b-4dd704c1e59c@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/c08d82f38ebf763b79bd43ae34b7310ee47aaacd
  • pgstattuple: Fix typo partitiond -> partitioned. Pointed out by Michael Paquier http://git.postgresql.org/pg/commitdiff/90e91e242fe99582b6d2dc18177e79d99c91695d

Andres Freund pushed:

  • Make simplehash.h grow hashtable in additional cases. Increase the size when either the distance between actual and optimal slot grows too large, or when too many subsequent entries would have to be moved. This addresses reports that the simplehash performed, sometimes considerably, worse than dynahash. The reason turned out to be that insertions into the hashtable where, due to the use of parallel query, in effect done from another hashtable, in hash-value order. If the target hashtable, due to mis-estimation, was sized a lot smaller than the source table(s) that lead to very imbalanced tables; a lot of entries in many close-by buckets from the source tables were inserted into a single, wider, bucket on the target table. As the growth factor was solely computed based on the fillfactor, the performance of the table decreased further and further. b81b5a96f424531b was an attempt to address this problem for hash aggregates (but not for bitmap scans), but it turns out that the current method of mixing hash values often actually leaves neighboring hash-values close to each other, just in different value range. It might be worth revisiting that independently of the performance issues addressed in this patch.. To address that problem resize tables in two additional cases: Firstly when the optimal position for an entry would be far from the actual position, secondly when many entries would have to be moved to make space for the new entry (while satisfying the robin hood property). Due to the additional resizing threshold it seems possible, and testing confirms that so far, that a higher fillfactor doesn't hurt performance and saves a bit of memory. It seems better to increase it now, before a release containing any of this code, rather than wonder in some later release. The various boundaries aren't determined in a particularly scientific manner, they might need some fine-tuning. In all my tests the new code now, even with parallelism, performs at least as good as the old code, in several scenarios significantly better. Reported-By: Dilip Kumar, Robert Haas, Kuntal Ghosh Discussion: https://postgr.es/m/CAFiTN-vagvuAydKG9VnWcoK=ADAhxmOa4ZTrmNsViBBooTnriQ@mail.gmail.com https://postgr.es/m/CAGz5QC+=fNTYgzMLTBUNeKt6uaWZFXJbkB5+7oWm-n9DwVxcLA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/d4c62a6b623d6eef88218158e9fa3cf974c6c7e5
  • Add amcheck extension to contrib. This is the beginning of a collection of SQL-callable functions to verify the integrity of data files. For now it only contains code to verify B-Tree indexes. This adds two SQL-callable functions, validating B-Tree consistency to a varying degree. Check the, extensive, docs for details. The goal is to later extend the coverage of the module to further access methods, possibly including the heap. Once checks for additional access methods exist, we'll likely add some "dispatch" functions that cover multiple access methods. Author: Peter Geoghegan, editorialized by Andres Freund Reviewed-By: Andres Freund, Tomas Vondra, Thomas Munro, Anastasia Lubennikova, Robert Haas, Amit Langote Discussion: CAM3SWZQzLMhMwmBqjzK+pRKXrNUZ4w90wYMUWfkeV8mZ3Debvw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/3717dc149ecf44b8be95350a68605ba7299474fd
  • amcheck: editorialize variable name & comment. No exclusive lock is taken anymore... http://git.postgresql.org/pg/commitdiff/fcd8d25d38b5f42ec4ae77a673813c2dc279ccf7
  • Enable 64 bit atomics on ARM64. Previously they were disabled due to performance concerns on 32bit arm, where 64bit atomics are often implemented via kernel traps. Author: Roman Shaposhnik Discussion: http://postgr.es/m/CA+ULb+uErkFuXUCCXWHYvnV5KnAyjGUzzRcPA-M0cgO+Hm4RSA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/f8f1430ae709c0018a217c77893754f38b3c5b93
  • Improve expression evaluation test coverage. Upcoming patches are revamping expression evaluation significantly. It therefore seems prudent to try to ensure that the coverage of the existing evaluation code is high. This commit adds coverage for the cases that can reasonably be tested. There's still a bunch of unreachable error messages and such, but otherwise this achieves nearly full regression test coverage (with the exception of the unused GetAttributeByNum/GetAttributeByName). Author: Andres Freund Discussion: https://postgr.es/m/20170310194021.ek4bs4bl2khxkmll@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/ce38949ba23ab311f274aa4196be09d18d30e5a6

Heikki Linnakangas pushed:

Magnus Hagander pushed:

Fujii Masao pushed:

  • Fix connection leak in DROP SUBSCRIPTION command, take 2. Commit 898a792eb8283e31efc0b6fcbc03bbcd5f7df667 fixed the connection leak issue, but it was an unreliable way of bugfix. This bugfix was assuming that walrcv_command() subroutine cannot throw an error, but it's untenable assumption. For example, if it will be changed so that an error is thrown, connection leak issue will happen again. This patch ensures that the connection is closed even when walrcv_command() subroutine throws an error. Patch by me, reviewed by Petr Jelinek and Michael Paquier Discussion: https://www.postgresql.org/message-id/2058.1487704345@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/77d21970ae19418f321e6a76ddf1a57ae999c77a
  • Prevent logical rep workers with removed subscriptions from starting. Any logical rep workers must have their subscription entries in pg_subscription. To ensure this, we need to prevent the launcher from starting new worker corresponding to the subscription that DROP SUBSCRIPTION command is removing. To implement this, previously LogicalRepLauncherLock was introduced and held until the end of transaction running DROP SUBSCRIPTION. But using LWLock for that purpose was not valid. Instead, this commit changes DROP SUBSCRIPTION so that it takes AccessExclusiveLock on pg_subscription, in order to ensure that the launcher cannot see any subscriptions being removed. Also this commit gets rid of LogicalRepLauncherLock. Patch by me, reviewed by Petr Jelinek Discussion: https://www.postgresql.org/message-id/CAHGQGwHPi8ky-yANFfe0sgmhKtsYcQLTnKx07bW9S7-Rn1746w@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4eafdcc27608dfb8a3afa3155d6ae07f66179782

Ãlvaro Herrera pushed:

  • Support XMLTABLE query expression. XMLTABLE is defined by the SQL/XML standard as a feature that allows turning XML-formatted data into relational form, so that it can be used as a <table primary> in the FROM clause of a query. This new construct provides significant simplicity and performance benefit for XML data processing; what in a client-side custom implementation was reported to take 20 minutes can be executed in 400ms using XMLTABLE. (The same functionality was said to take 10 seconds using nested PostgreSQL XPath function calls, and 5 seconds using XMLReader under PL/Python). The implemented syntax deviates slightly from what the standard requires. First, the standard indicates that the PASSING clause is optional and that multiple XML input documents may be given to it; we make it mandatory and accept a single document only. Second, we don't currently support a default namespace to be specified. This implementation relies on a new executor node based on a hardcoded method table. (Because the grammar is fixed, there is no extensibility in the current approach; further constructs can be implemented on top of this such as JSON_TABLE, but they require changes to core code.) Author: Pavel Stehule, Ãlvaro Herrera Extensively reviewed by: Craig Ringer Discussion: https://postgr.es/m/CAFj8pRAgfzMD-LoSmnMGybD0WsEznLHWap8DO79+-GTRAPR4qA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/fcec6caafa2346b6c9d3ad5065e417733bd63cd9
  • Fix XMLTABLE on older libxml2. libxml2 older than 2.9.1 does not have xmlXPathSetContextNode (released in 2013, so reasonable platforms have trouble). That function is fairly trivial, so I have inlined it in the one added caller. This passes tests on my machine; let's see what the buildfarm thinks about it. Per joint complaint from Tom Lane and buildfarm. http://git.postgresql.org/pg/commitdiff/a9f66f92533b2bfd7abf289219152091b7697e87

Michael Meskes pushed:

Joe Conway pushed:

Correctifs en attente

Jim Nasby sent in another revision of a patch to add SPI_execute_callback() and a callback-based DestReceiver, and modify plpython to use same.

Etsuro Fujita, Ashutosh Bapat, and David Rowley traded patches to fix an issue where foreign Join pushdowns were not working properly for outer joins.

Amit Langote sent in two more revisions of a patch to avoid creating scan nodes for partitioned tables and not allocate storage for partitioned tables.

Kyotaro HORIGUCHI sent in another revision of a patch to use a radix tree for character conversion.

Kyotaro HORIGUCHI sent in another revision of a patch to fix some infelicities between logical replication and database encoding.

Masahiko Sawada sent in a patch to update the documentation to reflect the fact that oldestxmin is now shown in VACUUM.

Dagfinn Ilmari Mannsåker sent in another revision of a patch to fix many perlcritic exceptions.

Kyotaro HORIGUCHI sent in a patch to add a WAL relief vent for replication slots.

Takeshi Ideriha sent in another revision of a patch to add DECLARE STATEMENT to ECPG.

Petr Jelínek sent in a patch to logical replication apply to run with sync commit off by default.

Robert Haas sent in a patch to fix the computation of parallel workers to distinguish "can't have a heap page" from "found zero heap pages."

Kyotaro HORIGUCHI sent in another revision of a patch to remove NamedLWLockTrancheArray.

Masahiko Sawada sent in another revision of a patch to add a GUC for cleanup indexes threshold.

Thomas Munro sent in two more revisions of a patch to add a [parallel [shared]] hash.

Aleksander Alekseev sent in three more revisions of a patch to make declarative partitioning work faster when large numbers of partitions are present.

Masahiko Sawada sent in another revision of a patch to report the number of skipped frozen pages by manual VACUUM.

David Steele sent in another revision of a patch to add a configurable file mode mask.

Amos Bird sent in a patch to make psql show indexes with type info.

Andreas Karlsson sent in a patch to rename the default log directory from pg_log to log.

Petr Jelínek and Peter Eisentraut traded patches to allow enable an existing data copy for logical replication.

Andres Freund sent in another revision of a patch to speed up expression processing.

Jan Michálek sent in another revision of a patch to allow formatting tables in psql as markdown, mediawiki, and rst.

Masahiko Sawada sent in another revision of a patch to allow transactions across foreign tables.

Masahiko Sawada sent in a patch to document the fact that PREPARE TRANSACTION is now part of ECPG.

Simon Riggs and Dean Rasheed traded patches to speed up replay of AccessExclusiveLock.

Tomas Vondra sent in another revision of a patch to implement two slab-like memory allocators.

David Rowley sent in a patch to fix some warnings in the slab-like memory allocators patch.

Haribabu Kommi sent in another revision of a patch to refactor handling of database attributes between pg_dump and pg_dumpall.

Haribabu Kommi sent in another revision of a patch to implement a pg_stat_sql view.

Pavan Deolasee sent in another revision of a patch to skip all-visible pages during second HeapScan of CREATE INDEX CONCURRENTLY.

Michaël Paquier sent in another revision of a patch to implement SASLprep.

Ivan Kartyshov sent in two more revisions of a patch to make async slave to wait for lsn to be replayed.

Michaël Paquier sent in two revisions of a patch to add clause PASSWORD (val USING protocol) to CREATE/ALTER ROLE.

Rafia Sabih and Robert Haas traded patches to enable parallelism for queries coming from SQL or other PL functions.

Daniel Vérité and Vaishnavi Prabakaran traded patches to add batch/pipelining support for libpq.

Takayuki Tsunakawa sent in a patch to refactor strinfo in dblink.

Daniel Vérité and Vaishnavi Prabakaran traded patches to add batch/pipelining support for libpq.

Takayuki Tsunakawa sent in a patch to refactor strinfo in dblink.

Amit Khandekar sent in two more revisions of a patch to implement parallel append.

Kevin Grittner sent in two more revisions of a patch to add transition table support for, among other things, statement triggers and materialized view refreshes.

Kuntal Ghosh sent in another revision of a patch to add WAL consistency check support for hash indexes.

Andrew Gierth sent in two more revisions of a patch to add hash support for grouping sets.

Takayuki Tsunakawa sent in another revision of a patch to support huge pages on Windows.

Craig Ringer sent in a patch to fix an off-by-one error in logical slot resource retention.

Rushabh Lathia sent in another revision of a patch to add wait events for disk I/O.

Amit Langote sent in a patch to use ALTER FOREIGN TABLE with foreign table in tests.

Ãlvaro Herrera sent in another revision of a patch to add WARM.

Victor Wagner sent in three more revisions of a patch to add explicit subtransactions for PL/Tcl.

Peter Eisentraut sent in another revision of a patch to cast result of copyObject() to correct type.

Peter Eisentraut sent in another revision of a patch to implement ICU integration.

Amit Langote sent in another revision of a patch to fix an infelicity between declarative partitioning and ON CONFLICT.

Amit Langote sent in another revision of a patch to improve the documents on declarative partitioning.

Amit Langote sent in a patch to fix a bug in get_partition_for_tuple.

Peter Geoghegan sent in a patch to fix the amcheck build on Windows.

Petr Jelínek sent in a patch to fix remote position tracking in logical replication.

Kuntal Ghosh sent in another revision of a patch to expose wait events for non-backends.

Beena Emerson sent in another revision of a patch to allow increasing the default WAL segment size.

Vinayak Pokale sent in another revision of a patch to add an ANALYZE command progress checker.

Michaël Paquier sent in two more revisions of a patch to fix unlogged flushes.

Surafel Temesgen and Pavel Stěhule traded patches to implement CORRESPONDING.

Tom Lane sent in two revisions of a patch to upgrade the postmaster's log messages about bind/listen errors.

Michaël Paquier sent in a patch to change references of password encryption to hashing.

Nikita Glukhov sent in another revision of a patch to fix a bug in SP-GiST box_ops.

Ashutosh Bapat sent in another revision of a patch to implement partition-wise join for join between (declaratively) partitioned tables.

Dmitry Dolgov sent in another revision of a patch to add FTS support for JSON[B].

Thomas Munro sent in another revision of a patch to fix SERIALIZABLE isolation for parallel query.

Nikhil Sontakke sent in another revision of a patch to speed up two-phase transactions.

Pavel Stěhule sent in a patch to add VERBOSE_SORT_COLUMNS and VERBOSE_SORT_DIRECTION to psql.

Pavel Stěhule sent in a patch to add default namespaces for XPath expressions.

Tom Lane and Andres Freund traded patches to speed up running all the tests.

Tom Lane sent in another revision of a patch to add \if and friends to psql.

Nikolay Shaplov sent in another revision of a patch to move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM.

Stephen Frost sent in two more revisions of a patch to add macaddr 64 bit (EUI-64) datatype support.

Peter Geoghegan sent in another revision of a patch to add parallel tuplesort (for parallel B-Tree index creation).

par chl le mardi 14 mars 2017 à 00h12

jeudi 9 mars 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 5 mars 2017

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170306025346.GA11640@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Andres Freund pushed:

  • Add "Slab" MemoryContext implementation for efficient equal-sized allocations. The default general purpose aset.c style memory context is not a great choice for allocations that are all going to be evenly sized, especially when those objects aren't small, and have varying lifetimes. There tends to be a lot of fragmentation, larger allocations always directly go to libc rather than have their cost amortized over several pallocs. These problems lead to the introduction of ad-hoc slab allocators in reorderbuffer.c. But it turns out that the simplistic implementation leads to problems when a lot of objects are allocated and freed, as aset.c is still the underlying implementation. Especially freeing can easily run into O(n^2) behavior in aset.c. While the O(n^2) behavior in aset.c can, and probably will, be addressed, custom allocators for this behavior are more efficient both in space and time. This allocator is for evenly sized allocations, and supports both cheap allocations and freeing, without fragmenting significantly. It does so by allocating evenly sized blocks via malloc(), and carves them into chunks that can be used for allocations. In order to release blocks to the OS as early as possible, chunks are allocated from the fullest block that still has free objects, increasing the likelihood of a block being entirely unused. A subsequent commit uses this in reorderbuffer.c, but a further allocator is needed to resolve the performance problems triggering this work. There likely are further potentialy uses of this allocator besides reorderbuffer.c. There's potential further optimizations of the new slab.c, in particular the array of freelists could be replaced by a more intelligent structure - but for now this looks more than good enough. Author: Tomas Vondra, editorialized by Andres Freund Reviewed-By: Andres Freund, Petr Jelinek, Robert Haas, Jim Nasby Discussion: https://postgr.es/m/d15dff83-0b37-28ed-0809-95a5cc7292ad@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/58b25e98106dbe062cec0f3d31d64977bffaa4af
  • Use the new "Slab" context for some allocations in reorderbuffer.h. Note that this change alone does not yet fully address the performance problems triggering this work, a large portion of the slowdown is triggered by the tuple allocator, which isn't converted to the new allocator. It would be possible to do so, but using evenly sized objects, like both the current implementation in reorderbuffer.c and slab.c, wastes a fair amount of memory. A later patch by Tomas will introduce a better approach. Author: Tomas Vondra Reviewed-By: Andres Freund Discussion: https://postgr.es/m/d15dff83-0b37-28ed-0809-95a5cc7292ad@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/9fab40ad32efa4038d19eaed975bb4c1713ccbc0
  • Make useful infrastructure from aset.c generally available. An upcoming patch introduces a new type of memory context. To avoid duplicating debugging infrastructure within aset.c, move useful pieces to memdebug.[ch]. While touching aset.c, fix printf format code in AllocFree* debug macros. Author: Tomas Vondra Reviewed-By: Andres Freund Discussion: https://postgr.es/m/b3b2245c-b37a-e1e5-ebc4-857c914bc747@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/bfd12cccbd72c1846bfa3e4031155c9bd479d70a
  • Overhaul memory management README. The README was written as a "historical account", and that style hasn't aged particularly well. Rephrase it to describe the current situation, instead of having various version specific comments. This also updates the description of how allocated chunks are associated with their corresponding context, the method of which has changed in the preceding commit. Author: Andres Freund Discussion: https://postgr.es/m/20170228074420.aazv4iw6k562mnxg@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/f4e2d50cd7483a068c0a32e56b2d40f980cdea72
  • Reduce size of common allocation header. The new slab allocator needs different per-allocation information than the classical aset.c. The definition in 58b25e981 wasn't sufficiently careful on 32 platforms with 8 byte alignment, leading to buildfarm failures. That's not entirely easy to fix by just adjusting the definition. As slab.c doesn't actually need the size part(s) of the common header, all chunks are equally sized after all, it seems better to instead reduce the header to the part needed by all allocators, namely which context an allocation belongs to. That has the advantage of reducing the overhead of slab allocations, and also allows for more flexibility in future allocators. To avoid spreading the logic about accessing a chunk's context around, centralize it in GetMemoryChunkContext(), which allows to delete a good number of lines. A followup commit will revise the mmgr/README portion about StandardChunkHeader, and more. Author: Andres Freund Discussion: https://postgr.es/m/20170228074420.aazv4iw6k562mnxg@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/7e3aa03b418d604d33040ed8fb866857dae82a02
  • Fix assertion failure due to over-eager code deduplication. In the previous commit I'd made MemoryContextContains() use GetMemoryChunkContext(), but that causes trouble when the passed pointer isn't allocated in any memory context - that's probably something we shouldn't do, but the previous commit isn't a place for a "policy" change. http://git.postgresql.org/pg/commitdiff/123ccbe58309d08e42009e99a4b34a3a1aef7798
  • Fix s/ITERTOR/ITERATOR/ typo in simplehash.h. This could lead to problem when simplehash.h is used to define two different types of hashtable visible in the same translation unit. Reported-By: Josh Soref Discussion: https://postgr.es/m/CACZqfqCC7WdBAY=rQePb9-qW1rjdaTdHsV5KoVejHkDb6qrtOg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/8f7277dfb5e703a034dbce7b155d998e577a6bc3
  • Fix two recently introduced grammar errors in mmgr/README. These were introduced by me in f4e2d50c. Reported-By: Tomas Vondra Discussion: https://postgr.es/m/11adca69-be28-44bc-a801-64e6d53851e3@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/1309375e706440e1e2fe2a5bb74effbc639261ef

Peter Eisentraut pushed:

Tom Lane pushed:

  • Remove PL/Tcl's "module" facility. PL/Tcl has long had a facility whereby Tcl code could be autoloaded from a database table named "pltcl_modules". However, nobody is using it, as evidenced by the recent discovery that it's never been fixed to work with standard_conforming_strings turned on. Moreover, it's rather shaky from a security standpoint, and the table design is very old and crufty (partly because it dates from before we had TOAST). A final problem is that because the table-population scripts depend on the Tcl client library Pgtcl, which we removed from the core distribution in 2004, it's impossible to create a self-contained regression test for the feature. Rather than try to surmount these problems, let's just remove it. A follow-on patch will provide a way to execute user-defined initialization code, similar to features that exist in plperl and plv8. With that, it will be possible to implement this feature or similar ones entirely in userspace, which is where it belongs. Discussion: https://postgr.es/m/22067.1488046447@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/817f2a586342767d3289a320bb1dac5dcbb76979
  • Allow index AMs to return either HeapTuple or IndexTuple format during IOS. Previously, only IndexTuple format was supported for the output data of an index-only scan. This is fine for btree, which is just returning a verbatim index tuple anyway. It's not so fine for SP-GiST, which can return reconstructed data that's much larger than a page. To fix, extend the index AM API so that index-only scan data can be returned in either HeapTuple or IndexTuple format. There's other ways we could have done it, but this way avoids an API break for index AMs that aren't concerned with the issue, and it costs little except a couple more fields in IndexScanDescs. I changed both GiST and SP-GiST to use the HeapTuple method. I'm not very clear on whether GiST can reconstruct data that's too large for an IndexTuple, but that seems possible, and it's not much of a code change to fix. Per a complaint from Vik Fearing. Reviewed by Jason Li. Discussion: https://postgr.es/m/49527f79-530d-0bfe-3dad-d183596afa92@2ndquadrant.fr http://git.postgresql.org/pg/commitdiff/9b88f27cb42fe8ff59ddc75e29c005624b8850a2
  • Update documentation of tsquery_phrase(). Missed in commit 028350f61. Noted by Eiji Seki. http://git.postgresql.org/pg/commitdiff/d99706ed5178d7f37ac322e02e8c56e4e5e0e99a
  • In rebuild_relation(), don't access an already-closed relcache entry. This reliably fails with -DRELCACHE_FORCE_RELEASE, as reported by Andrew Dunstan, and could sometimes fail in normal operation, resulting in a wrong persistence value being used for the transient table. It's not immediately clear to me what effects that might have beyond the risk of a crash while accessing OldHeap->rd_rel->relpersistence, but it's probably not good. Bug introduced by commit f41872d0c, and made substantially worse by commit 85b506bbf, which added a second such access significantly later than the heap_close. I doubt the first reference could fail in a production scenario, but the second one definitely could. Discussion: https://postgr.es/m/7b52f900-0579-cda9-ae2e-de5da17090e6@2ndQuadrant.com http://git.postgresql.org/pg/commitdiff/dbca84f04ed5debe748029699aa44fa86beca32d

Robert Haas pushed:

  • hash: Refactor bucket squeeze code. In preparation for adding write-ahead logging to hash indexes, refactor _hash_freeovflpage and _hash_squeezebucket so that all related page modifications happen in a single section of code. The previous coding assumed that it would be fine to move tuples one at a time, and also that the various operations involved in freeing an overflow page didn't necessarily all need to be done together, all of which is true if you don't care about write-ahead logging. Amit Kapila, with slight changes by me. http://git.postgresql.org/pg/commitdiff/b0f18cb77f50a54e997d857d592f6a511617f52c
  • hash: Refactor overflow page allocation. As with commit b0f18cb77f50a54e997d857d592f6a511617f52c, the goal here is to move all of the related page modifications to a single section of code, in preparation for adding write-ahead logging. Amit Kapila, with slight changes by me. The larger patch series of which this is a part has been reviewed and tested by Ãlvaro Herrera, Ashutosh Sharma, Mark Kirkwood, Jeff Janes, and Jesper Pedersen, all of whom should also have been credited in the previous commit message. http://git.postgresql.org/pg/commitdiff/30df93f698d016d086e8961aa6c6076b37ea0ef4
  • hash: Refactor and clean up bucket split code. As with commit 30df93f698d016d086e8961aa6c6076b37ea0ef4 and commit b0f18cb77f50a54e997d857d592f6a511617f52c, the goal here is to move all of the related page modifications to a single section of code, in preparation for adding write-ahead logging. Amit Kapila, with slight changes by me. The larger patch series of which this is a part has been reviewed and tested by Ãlvaro Herrera, Ashutosh Sharma, Mark Kirkwood, Jeff Janes, and Jesper Pedersen. http://git.postgresql.org/pg/commitdiff/21a3cf41284c08307ef9abe3400be5dc53723519
  • Update comments overlooked by 2f5c9d9c9cec436e55847ec580606d7e88067df6. Tomas Vondra http://git.postgresql.org/pg/commitdiff/fa42b2005f0cd825fe5a5fd4db93a7c30b5fe883
  • Don't uselessly rewrite, truncate, VACUUM, or ANALYZE partitioned tables. Also, recursively perform VACUUM and ANALYZE on partitions when the command is applied to a partitioned table. In passing, some related documentation updates. Amit Langote, reviewed by Michael Paquier, Ashutosh Bapat, and by me. Discussion: http://postgr.es/m/47288cf1-f72c-dfc2-5ff0-4af962ae5c1b@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/3c3bb99330aa9b4c2f6258bfa0265d806bf365c3
  • Refactor bitmap heap scan in preparation for parallel support. The final patch will be less messy if the prefetching support is a bit better isolated, so do that. Dilip Kumar, with some changes by me. The larger patch set of which this is a part has been reviewed and tested by (at least) Andres Freund, Amit Khandekar, Tushar Ahuja, Rafia Sabih, Haribabu Kommi, and Thomas Munro. http://git.postgresql.org/pg/commitdiff/9e0fe09fc5dc1135479b532d1806e28cbc5a35aa
  • Improve error reporting for tuple-routing failures. Currently, the whole row is shown without column names. Instead, adopt a style similar to _bt_check_unique() in ExecFindPartition() and show the failing key: (key1, ...) = (val1, ...). Amit Langote, per a complaint from Simon Riggs. Reviewed by me; I also adjusted the grammar in one of the comments. Discussion: http://postgr.es/m/9f9dc7ae-14f0-4a25-5485-964d9bfc19bd@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/5a73e17317e91912b2755f7960d5bf31d374cf31
  • Notify bgworker registrant after freeing worker slot. Tom Lane observed buildfarm failures caused by the select_parallel regression test trying to launch new parallel queries before the worker slots used by the previous ones were freed. Try to fix this by having the postmaster free the worker slots before it sends the SIGUSR1 notifications to the registering process. This doesn't completely eliminate the possibility that the user backend might (correctly) observe the worker as dead before the slot is free, but I believe it should make the window significantly narrower. Patch by me, per complaint from Tom Lane. Reviewed by Amit Kapila. Discussion: http://postgr.es/m/30673.1487310734@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/aea5d298362e881b13d95a48c5ae116879237389
  • Add pg_current_logfile() function. The syslogger will write out the current stderr and csvlog names, if it's running and there are any, to a new file in the data directory called "current_logfiles". We take care to remove this file when it might no longer be valid (but not at shutdown). The function pg_current_logfile() can be used to read the entries in the file. Gilles Darold, reviewed and modified by Karl O. Pinc, Michael Paquier, and me. Further review by Ãlvaro Herrera and Christoph Berg. http://git.postgresql.org/pg/commitdiff/19dc233c32f2900e57b8da4f41c0f662ab42e080

Magnus Hagander pushed:

Ãlvaro Herrera pushed:

Noah Misch pushed:

  • Handle unaligned SerializeSnapshot() buffer. Likewise in RestoreSnapshot(). Do so by copying between the user buffer and a stack buffer of known alignment. Back-patch to 9.6, where this last applies cleanly. In master, the select_parallel test dies with SIGBUS on "Oracle Solaris 10 1/13 s10s_u11wos_24a SPARC", building 32-bit with gcc 4.9.2. In 9.6 and 9.5, the buffers in question happen to be sufficiently-aligned, and this change is mere insurance against future 9.6 changes or extension code compromising that. http://git.postgresql.org/pg/commitdiff/7f3112135eb67e5df56cd34b8ce662bf6a2390e9

Simon Riggs pushed:

Correctifs en attente

Thomas Munro sent in two revisions of a patch to add a test for SSI SLRU wraparound.

Beena Emerson sent in another revision of a patch to allow setting the default WAL segment size at initdb time.

Kyotaro HORIGUCHI and Peter Eisentraut traded patches to fix an issue in the logical replication protocol.

Amit Langote sent in a patch to allow dropping declaratively partitioned table without CASCADE.

Noah Misch sent in a patch to use wrappers of PG_DETOAST_DATUM_PACKED() more.

Etsuro Fujita sent in a patch to add support for parameterized foreign joins.

Dagfinn Ilmari Mannsåker sent in a patch to add max_pred_locks_per_{relation,page} reloptions.

Alexander Korotkov sent in another revision of a patch to add incremental sort.

Haribabu Kommi sent in another revision of a patch to add infrastructure which will be used to parallelize DML and utility commands.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Ãlvaro Herrera sent in a patch to make a BRIN page range unsummarized.

Andrew Dunstan sent in another revision of a patch to enable tree_gin and btree_gist for enums.

Simon Riggs sent in a patch to reduce the amount of bloat resulting from running CREATE INDEX CONCURRENTLY by destroying the snapshot taken in the first phase, before entering the second phase.

Kyotaro HORIGUCHI sent in a patch to make slotSegNo a XLogSegNo, which it already should have been.

Daisuke Higuchi sent in a patch to enable walsender for async to wait until walsender for sync confirms that WAL is written to disk, controlled by a new boolean GUC, async_walsender_delay.

Robert Haas, Amit Langote, and Ashutosh Bapat traded patches to fix an infelicity between declaratively partitioned tables and relfilenode.

Kyotaro HORIGUCHI sent in two more revisions of a patch to implement character conversion with a radix tree.

Takayuki Tsunakawa sent in a PoC patch to implement statement-level ROLLBACK.

Kyotaro HORIGUCHI sent in a patch to add WAL relief vent for replication slots.

Surafel Temesgen sent in a patch to disallow multiple queries per PQexec().

Tom Lane sent in two revisions of a patch to remove PL/Tcl's unused modules infrastructure.

Kuntal Ghosh sent in a patch to implement WAL consistency checking for hash indexes.

David Steele sent in two revisions of a patch to make pg_stop_backup() archive wait optional.

Dmitry Dolgov sent in a patch to add some full text search functions for JSONB.

Etsuro Fujita sent in a patch to the PostgreSQL FDW to enable evaluating placeholdervars on the remote server.

Simon Riggs sent in a patch to avoid bloat in CREATE INDEX CONCURRENTLY.

Dmitry Dolgov sent in another revision of a patch to implement generic type subscripting.

David Steele sent in a patch to implement a configurable file mode mask.

Haribabu Kommi sent in a patch to refactor the handling of database attributes between pg_dump and pg_dumpall.

Peter Eisentraut sent in another revision of a patch to implement IDENTITY columns.

Peter Eisentraut sent in another revision of a patch to implement background sessions.

Peter Eisentraut sent in a patch to use the SQL standard error code for nextval.

Peter Eisentraut sent in another revision of a patch to cast result of copyObject() to the correct type.

Venkata B Nagothi sent in two more revisions of a patch to add recovery_start_point and recovery_incomplete.

Amit Kapila sent in another revision of a patch to add WAL logging for hash indexes.

Peter Moser sent in another revision of a patch to add temporal query processing with range types.

Peter Eisentraut sent in another revision of a patch to do some of the infrastructure for making PostgreSQL's source C++-compatible.

Peter Eisentraut sent in a patch to refactor dblink, replacing some macros with static functions.

Peter Eisentraut sent in a patch to make much of the Perl code perlcritic-clean.

Andres Freund sent in a patch to move contrib/seg to only use V1 calling conventions and remove support for version-0 calling conventions.

KaiGai Kohei sent in two more revisions of a patch to add a PassDownLimitBound for ForeignScan/CustomScan.

Tomas Vondra sent in a patch to update comments about CatalogUpdateIndexes.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Jim Nasby sent in another revision of a patch to add SPI_execute_callback() and callback-based DestReceiver and minimally use same in PL/PythonU.

Ãlvaro Herrera sent in a patch to add autovacuum work items for BRIN summarization.

Brandur Leach sent in another revision of a patch to add SortSupport for the macaddr type.

Rahila Syed sent in a patch to add support for default partition in declarative partitioning.

Peter Eisentraut sent in another revision of a patch to enable referring to functions without arguments when the name is unique.

Corey Huinker sent in three more revisions of a patch to implement \if and friends in psql.

Yugo Nagata and Amul Sul traded different implementations of patches to implement hash partitioning for declaratively partitioned tables.

Vinayak Pokale sent in a patch to add a progress checker for ANALYZE.

Oleg Bartunov sent in a patch to implement SQL/JSON.

Pavan Deolasee sent in a patch to skip all-visible pages during second HeapScan of CREATE INDEX CONCURRENTLY.

Andres Freund sent in a patch to make simplehash.h grow hashtable in additional cases.

Dilip Kumar and Robert Haas traded patches to implement parallel merge join.

Lucas Fitti sent in a patch to use $ parameters as replacement characters for pg_stat_statements.

Jan Michálek sent in two more revisions of a patch to implement markdown, rst and mediawiki output formats in psql.

Andreas Karlsson sent in another revision of a patch to implement REINDEX CONCURRENTLY.

Anastasia Lubennikova sent in another revision of a patch to add covering + unique indexes.

Ãlvaro Herrera sent in another revision of a patch to improve BRIN cost estimates.

Dagfinn Ilmari Mannsåker sent in a patch to address all the ## no critic annotations except RequireFilenameMatchesPackage.

David Rowley sent in a patch to improve performance of replay of AccessExclusiveLock.

Ashutosh Bapat sent in two more revisions of a patch to improve partition-wise join between declaratively partitioned tables.

Michaël Paquier sent in two more revisions of a patch to implement SCRAM authentication.

Ãlvaro Herrera sent in another revision of a patch to optimize memory allocation in the 'bringetbitmap' function.

Tomas Vondra sent in two more revisions of a patch to implement extended statistics, formerly multivariate statistics.

Jim Mlodgenski sent in two revisions of a patch to track stats for materialized views.

Thomas Munro sent in another revision of a patch to measure replay lag.

Petr Jelínek sent in another revision of a patch to reserve global xmin for create slot snasphot export, not use on disk snapshots for snapshot export in logical decoding, prevent snapshot builder xmin from going backwards, fix xl_running_xacts usage in snapshot builder, and skip unnecessary snapshot builds.

Amit Langote sent in another revision of a patch to improve the documents for partitioning.

Ronan Dunklau sent in a patch to fix mergeappend costsort estimates.

Masahiko Sawada sent in a patch to implement 2PC in ECPG.

Rushabh Lathia sent in a patch to print the correct startup cost for the group aggregate.

Ãlvaro Herrera and Pavel StÄ›hule traded patches to implement xmltable().

Kyotaro HORIGUCHI sent in a patch to remove NamedLWLockTrancheArray.

Petr Jelínek sent in a patch to reorder the asynchronous libpq calls for replication connection.

Tomas Vondra sent in another revision of a patch to add page_checksum and bt_page_items(bytea).

Ashutosh Sharma sent in another revision of a patch to fix an issue where parallel seq. plan was not coming against inheritance or partition table.

David Rowley sent in a patch to fix an issue where foreign join pushdowns not working properly for outer joins.

par chl le jeudi 9 mars 2017 à 00h09

jeudi 2 mars 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 26 février 2017

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170226230150.GA2560@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Fix documentation of to_char/to_timestamp TZ, tz, OF formatting patterns. These are only supported in to_char, not in the other direction, but the documentation failed to mention that. Also, describe TZ/tz as printing the time zone "abbreviation", not "name", because what they print is elsewhere referred to that way. Per bug #14558. http://git.postgresql.org/pg/commitdiff/10257fc5ff74487a46594bd8c8c041878f409c17
  • Improve error message for misuse of TZ, tz, OF formatting patterns. Be specific about which pattern is being complained of, and avoid saying "it's not supported in to_date", which is just confusing if the error is actually coming out of to_timestamp. We can phrase it as "is only supported in to_char", instead. Also, use the term "formatting field" not "format pattern", because other error messages in the same file prefer that terminology. (This isn't terribly consistent with the documentation, so maybe we should change all these error messages?) http://git.postgresql.org/pg/commitdiff/1c073505e8e4fa8a03312fea714da25ab83cb5fa
  • Reject too-old Python versions a bit sooner. Commit 04aad4018 added this check after the search for a Python shared library, which seems to me to be a pretty unfriendly ordering. The search might fail for what are basically version-related reasons, and in such a case it'd be better to say "your Python is too old" than "could not find shared library for Python". http://git.postgresql.org/pg/commitdiff/4e5ce3c1aeadf81b504bc9d683b67950bd3f8766
  • Use less-generic table name in new regression test case. Creating global objects named "foo" isn't an especially wise thing, but especially not in a test script that has already used that name for something else, and most especially not in a script that runs in parallel with other scripts that use that name :-( Per buildfarm. http://git.postgresql.org/pg/commitdiff/1c95f0b478a91b58391720dcda35bc032e582564
  • Fix sloppy handling of corner-case errors in fd.c. Several places in fd.c had badly-thought-through handling of error returns from lseek() and close(). The fact that those would seldom fail on valid FDs is probably the reason we've not noticed this up to now; but if they did fail, we'd get quite confused. LruDelete and LruInsert actually just Assert'd that lseek never fails, which is pretty awful on its face. In LruDelete, we indeed can't throw an error, because that's likely to get called during error abort and so throwing an error would probably just lead to an infinite loop. But by the same token, throwing an error from the close() right after that was ill-advised, not to mention that it would've left the LRU state corrupted since we'd already unlinked the VFD from the list. I also noticed that really, most of the time, we should know the current seek position and it shouldn't be necessary to do an lseek here at all. As patched, if we don't have a seek position and an lseek attempt doesn't give us one, we'll close the file but then subsequent re-open attempts will fail (except in the somewhat-unlikely case that a FileSeek(SEEK_SET) call comes between and allows us to re-establish a known target seek position). This isn't great but it won't result in any state corruption. Meanwhile, having an Assert instead of an honest test in LruInsert is really dangerous: if that lseek failed, a subsequent read or write would read or write from the start of the file, not where the caller expected, leading to data corruption. In both LruDelete and FileClose, if close() fails, just LOG that and mark the VFD closed anyway. Possibly leaking an FD is preferable to getting into an infinite loop or corrupting the VFD list. Besides, as far as I can tell from the POSIX spec, it's unspecified whether or not the file has been closed, so treating it as still open could be the wrong thing anyhow. I also fixed a number of other places that were being sloppy about behaving correctly when the seekPos is unknown. Also, I changed FileSeek to return -1 with EINVAL for the cases where it detects a bad offset, rather than throwing a hard elog(ERROR). It seemed pretty inconsistent that some bad-offset cases would get a failure return while others got elog(ERROR). It was missing an offset validity check for the SEEK_CUR case on a closed file, too. Back-patch to all supported branches, since all this code is fundamentally identical in all of them. Discussion: https://postgr.es/m/2982.1487617365@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/f97de05a14bbd26cf0252906b44643e8923bdf85
  • Suppress unused-variable warning. Rearrange so we don't have an unused variable in disable-cassert case. Discussion: https://postgr.es/m/CAMkU=1x63f2QyFTeas83xJqD+Hm1PBuok1LrzYzS-OngDzYOVA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/c56ac2913a1f3adce674a2eb27257d0bca81317f
  • Fix contrib/pg_trgm's extraction of trigrams from regular expressions. The logic for removing excess trigrams from the result was faulty. It intends to avoid merging the initial and final states of the NFA, which is necessary, but in testing whether removal of a specific trigram would cause that, it failed to consider the combined effects of all the state merges that that trigram's removal would cause. This could result in a broken final graph that would never match anything, leading to GIN or GiST indexscans not finding anything. To fix, add a "tentParent" field that is used only within this loop, and set it to show state merges that we are tentatively going to do. While examining a particular arc, we must chase up through tentParent links as well as regular parent links (the former can only appear atop the latter), and we must account for state init/fin flag merges that haven't actually been done yet. To simplify the latter, combine the separate init and fin bool fields into a bitmap flags field. I also chose to get rid of the "children" state list, which seems entirely inessential. Per bug #14563 from Alexey Isayko, which the added test cases are based on. Back-patch to 9.3 where this code was added. Report: https://postgr.es/m/20170222111446.1256.67547@wrigleys.postgresql.org Discussion: https://postgr.es/m/8816.1487787594@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/9e43e8714c9e976e41b7429fa7c426c9a6e5e8e6
  • De-support floating-point timestamps. Per discussion, the time has come to do this. The handwriting has been on the wall at least since 9.0 that this would happen someday, whenever it got to be too much of a burden to support the float-timestamp option. The triggering factor now is the discovery that there are multiple bugs in the code that attempts to implement use of integer timestamps in the replication protocol even when the server is built for float timestamps. The internal float timestamps leak into the protocol fields in places. While we could fix the identified bugs, there's a very high risk of introducing more. Trying to build a wall that would positively prevent mixing integer and float timestamps is more complexity than we want to undertake to maintain a long-deprecated option. The fact that these bugs weren't found through testing also indicates a lack of interest in float timestamps. This commit disables configure's --disable-integer-datetimes switch (it'll still accept --enable-integer-datetimes, though), removes direct references to USE_INTEGER_DATETIMES, and removes discussion of float timestamps from the user documentation. A considerable amount of code is rendered dead by this, but removing that will occur as separate mop-up. Discussion: https://postgr.es/m/26788.1487455319@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/b6aa17e0ae367afdcea07118e016111af4fa6bc3
  • Remove pg_control's enableIntTimes field. We don't need it any more. pg_controldata continues to report that date/time type storage is "64-bit integers", but that's now a hard-wired behavior not something it sees in the data. This avoids breaking pg_upgrade, and perhaps other utilities that inspect pg_control this way. Ditto for pg_resetwal. I chose to remove the "bigint_timestamps" output column of pg_control_init(), though, as that function hasn't been around long and probably doesn't have ossified users. Discussion: https://postgr.es/m/26788.1487455319@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/d28aafb6dda326688e2f042c95c93ea57963c03c
  • Remove now-dead code for !HAVE_INT64_TIMESTAMP. This is a basically mechanical removal of #ifdef HAVE_INT64_TIMESTAMP tests and the negative-case controlled code. Discussion: https://postgr.es/m/26788.1487455319@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/b9d092c962ea3262930e3c31a8c3d79b66ce9d43
  • Consistently declare timestamp variables as TimestampTz. Twiddle the replication-related code so that its timestamp variables are declared TimestampTz, rather than the uninformative "int64" that was previously used for meant-to-be-always-integer timestamps. This resolves the int64-vs-TimestampTz declaration inconsistencies introduced by commit 7c030783a, though in the opposite direction to what was originally suggested. This required including datatype/timestamp.h in a couple more places than before. I decided it would be a good idea to slim down that header by not having it pull in <float.h> etc, as those headers are no longer at all relevant to its purpose. Unsurprisingly, a small number of .c files turn out to have been depending on those inclusions, so add them back in the .c files as needed. Discussion: https://postgr.es/m/26788.1487455319@sss.pgh.pa.us Discussion: https://postgr.es/m/27694.1487456324@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c29aff959dc64f7321062e7f33d8c6ec23db53d3
  • Add an Assert that enum_cmp_internal() gets passed an FmgrInfo pointer. If someone were to try to call one of the enum comparison functions using DirectFunctionCallN, it would very likely seem to work, because only in unusual cases does enum_cmp_internal() need to access the typcache. But once such a case occurred, code like that would crash with a null pointer dereference. To make an oversight of that sort less likely to escape detection, add a non-bypassable Assert that fcinfo->flinfo isn't NULL. Discussion: https://postgr.es/m/25226.1487900067@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/6d493e1a013514a6f0abb5d30d08219c1831cfec
  • Suppress compiler warnings in ecpg test on newer Windows toolchains. nan_test.pgc supposed that it could unconditionally #define isnan() and isinf() on WIN32. This was evidently copied at some point from src/include/port/win32.h, but nowadays there's a test on _MSC_VER there. Make nan_test.pgc look the same. Per buildfarm warnings. There's no evidence this produces anything worse than a warning, and besides it's only a test case, so I don't feel a need to back-patch. http://git.postgresql.org/pg/commitdiff/c5658a0764d5ac5ea8c2c11d27c62d5472234227
  • Fix unportable definition of BSWAP64() macro. We have a portable way of writing uint64 constants, but whoever wrote this macro didn't know about it. While at it, fix unsafe under-parenthesization of arguments. That might be moot, because there are already good reasons not to use the macro on anything more complicated than a simple variable, but it's still poor practice. Per buildfarm warnings. http://git.postgresql.org/pg/commitdiff/41c16edcf6c90d1f42810ea523b7e65c99edad50
  • Remove useless duplicate inclusions of system header files. c.h #includes a number of core libc header files, such as <stdio.h>. There's no point in re-including these after having read postgres.h, postgres_fe.h, or c.h; so remove code that did so. While at it, also fix some places that were ignoring our standard pattern of "include postgres[_fe].h, then system header files, then other Postgres header files". While there's not any great magic in doing it that way rather than system headers last, it's silly to have just a few files deviating from the general pattern. (But I didn't attempt to enforce this globally, only in files I was touching anyway.) I'd be the first to say that this is mostly compulsive neatnik-ism, but over time it might save enough compile cycles to be useful. http://git.postgresql.org/pg/commitdiff/9e3755ecb2d058f7d123dd35a2e1784006190962
  • Remove some configure header-file checks that we weren't really using. We had some AC_CHECK_HEADER tests that were really wastes of cycles, because the code proceeded to #include those headers unconditionally anyway, in all or a large majority of cases. The lack of complaints shows that those headers are available on every platform of interest, so we might as well let configure run a bit faster by not probing those headers at all. I suspect that some of the tests I left alone are equally useless, but since all the existing #includes of the remaining headers are properly guarded, I didn't touch them. http://git.postgresql.org/pg/commitdiff/2bd7f85796ec373ecae61dd480437b3e668ec883
  • Put back #include <windows.h> in dirmod.c. I removed this in commit 9e3755ecb, reasoning that the win32.h port-specific header file included by c.h would have provided it. However, that's only true on native win32 builds, not Cygwin builds. It may be that some of the other <windows.h> inclusions also need to be put back on the same grounds; but this is the only one that is clearly meant to be included #ifdef __CYGWIN__, so maybe this is the extent of the problem. Awaiting further buildfarm results. http://git.postgresql.org/pg/commitdiff/285ca26132abdd0a1adc11a21789f103c4e3f6d8

Simon Riggs pushed:

Peter Eisentraut pushed:

Fujii Masao pushed:

Ãlvaro Herrera pushed:

Robert Haas pushed:

Andrew Dunstan pushed:

  • Correctly handle array pseudotypes in to_json and to_jsonb. Columns with array pseudotypes have not been identified as arrays, so they have been rendered as strings in the json and jsonb conversion routines. This change allows them to be rendered as json arrays, making it possible to deal correctly with the anyarray columns in pg_stats. http://git.postgresql.org/pg/commitdiff/502a3832cc54c7115dacb8a2dae06f0620995ac6

Bruce Momjian pushed:

  • pg_upgrade docs: clarify instructions on standby extensions. Previously the pg_upgrade standby upgrade instructions said not to execute pgcrypto.sql, but it should have referenced the extension command "CREATE EXTENSION pgcrypto". This patch makes that doc change. Reported-by: a private bug report Backpatch-through: 9.4, where standby instructions were added http://git.postgresql.org/pg/commitdiff/5639ceddcb7f3efa8751b2ba6e50cc1d27cc2a45

Magnus Hagander pushed:

Correctifs en attente

Rushabh Lathia sent in another revision of a patch to implement Gather Merge.

Michaël Paquier sent in another revision of a patch to make hba configuration for SASL more extensible and implement SASLprep aka NFKC for SCRAM authentication.

Amit Langote and Ashutosh Bapat traded patches to allow dropping partitioned table without CASCADE.

Rushabh Lathia sent in another revision of a patch to implement wait events for disk I/O.

Dave Page sent in a patch to implement pg_ls_wal_dir() and pg_ls_logdir().

Tomas Vondra sent in a patch to add page_checksum and bt_page_items(bytea) to the pageinspect supplied module.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Takeshi Ideriha sent in another revision of a patch to allow DECLARE STATEMENT to set up a connection in ECPG.

Michaël Paquier sent in another revision of a patch to add tab completion in psql for the new SUBSCRIPTION commands.

Thomas Munro sent in another revision of a patch to make SERIALIZABLE work with parallel query.

Etsuro Fujita and Rushabh Lathia traded patches to push down more UPDATEs/DELETEs in postgres_fdw.

Petr Jelínek sent in a patch to smooth the transition from floating point timestamps.

Amit Langote and Ashutosh Bapat traded patches to take note of the fact that partitioned tables are empty themselves by preventing the from trying to do things to them that need to access files, not allocating storage for partitioned tables, and avoiding creating scan nodes for partitioned tables.

Simon Riggs sent in two revisions of a patch to ensure that SnapshotResetXmin() is not issued at the end of xact and reduce the calls to SnapshotResetXmin() using a simple heuristic to reduce the effects.

Beena Emerson sent in two more revisions of a patch to allow increasing the default WAL segment size.

Peter Moser sent in another revision of a patch to add ALIGN and NORMALIZE operators for temporal queries.

Tomas Vondra and Andres Freund traded patches to add two slab-like memory allocators.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Jim Mlodgenski sent in a patch to add system views for monitoring materialized views.

Amit Langote sent in a patch to show only the partition key upon failing to find a partition.

Mithun Cy sent in another revision of a patch to expand hash indexes differently.

Tatsuo Ishii sent in a patch to improve the calculation of statement_timeout for the extended query protocol.

Amit Langote sent in a patch to add regression tests foreign partition DDL.

Vaishnavi Prabakaran sent in another revision of a patch to add batch/pipelining support to libpq.

Masahiko Sawada sent in two more revisions of a patch to fix an infelicity between DROP SUBSCRIPTION and ROLLBACK.

Thomas Munro sent in two more revisions of a patch to measure replication lag.

Petr Jelínek sent in three more revisions of a patch to fix some snapbuild woes.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Robins Tharakan and Simon Riggs traded patches to make pg_dumpall work without access to pg_authid.

Rafia Sabih sent in two revisions of a patch to make parallelism work for queries coming from functions (SQL and several PLs).

Michael Banck sent in two revisions of a patch to reorder tablespaces in basebackup tar stream for backup_label.

Petr Jelínek sent in two more revisions of a patch to use asynchronous connect API in libpqwalreceiver, fix after trigger execution in logical replication, and add RENAME support for PUBLICATIONs and SUBSCRIPTIONs.

Pavan Deolasee sent in a patch to remove all direct references to ip_posid and ip_blkid members of ItemPointerData struct and instead use ItemPointerGetOffsetNumber and ItemPointerGetBlockNumber macros respectively to access these members.

Peter Eisentraut sent in a patch to silence compiler warnings from gcc -O3.

Nikolay Shaplov sent in another revision of a patch to move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM.

Jim Nasby sent in a patch to move refreshes of materialized views to last in dbObjectTypePriority[].

Jim Nasby sent in a patch to standardize on one of objsubid, subobjid.

Takayuki Tsunakawa sent in another revision of a patch to support huge pages on Windows.

Kyotaro HORIGUCHI sent in two more revisions of a patch to implement asynchronous execution.

Yugo Nagata sent in two more revisions of a patch to fix a comment on the JunkFilter struct.

Bernd Helmle sent in a patch to make subquery alias optional in FROM clause.

Chhatoi Pritam Baral sent in a patch to make the planner expand "foocol <@ 'x,y'::foorange" into "foocol between x and y".

Petr Jelínek sent in two more revisions of a patch to add logical replication support for initial data copy.

Mithun Cy sent in another revision of a patch to implement auto_prewarm.

Corey Huinker sent in four more revisions of a patch to add \if and friends to psql.

Stephen Frost sent in another revision of a patch to fix pg_upgrade blob comments.

Masahiko Sawada sent in a patch to report the number of skipped frozen pages by manual VACUUM.

Craig Ringer sent in a patch to better detect timeouts in PostgresNode::psql using regex.

Eiji Seki sent in another revision of a patch to add a GetOldestXminExtend for ignoring arbitrary vacuum flags.

Haribabu Kommi sent in a patch to make it possible to have utility commands benefit from a parallel plan.

Tom Lane sent in a patch to turn AllocBlockData from a linked list into a doubly-linked list.

Andrew Dunstan sent in a patch to make enums indexable in the btree_gin and btree_gist extensions.

Tom Lane sent in a patch to assert that an enum comparator can use cache.

Dave Page sent in a patch to add a default role called pg_monitor, complete with a lot of things such a role should have.

Naoki Okano sent in a patch to add an --enable-parse-comment option to ECPG.

Pavel Stěhule sent in a patch to add a VERBOSE_SORT variable to psql.

Andrew Gierth sent in another revision of a patch to add hash support for grouping sets.

Dilip Kumar sent in two more revisions of a patch to implement parallel merge join.

David Rowley sent in a patch to add recognition of "range queries" like "x > 34 AND x < 42" and IS [NOT] NULL to clausesel.

Simon Riggs sent in another revision of a patch to make some changes to the recovery.conf API.

Peter Eisentraut sent in another revision of a patch to enable DROP FUNCTION to drop multiple functions at once.

Peter Eisentraut sent in a patch to add cursor and execute methods to plan object in PL/PythonU.

Simon Riggs sent in another revision of a patch to reduce lock levels for reloptions.

Andrew Dunstan sent in two revisions of a patch to turn the DirectFunctionCall{n]Coll functions into macros calling these functions and passing NULL as the flinfo param.

Petr Jelínek sent in a patch to prevent pg_dump -t from dumping publications.

par chl le jeudi 2 mars 2017 à 00h33

mercredi 1 mars 2017

Damien Clochard

3 Meetups PostgreSQL en 1 semaine !

Toulouse, Nantes ou Paris ? Vous avez l’embarras du choix 3 rendez-vous autour de PostgreSQL s’offrent à vous dans les jours à venir !

A noter pour être complet qu’il y a aussi un groupe actif à Lyon et un autre à Lille

Et si vous en voulez encore, vous avez aussi deux conférences PostgreSQL en approche :

  • Le PG Day Paris le 23 mars avec un programme très riche et intégralement en anglais

  • Le PG Day France le 8 juin chez Météo France à Toulouse

Bref il y en a pour tous les gouts et aux 4 coins de la France :-)

par Damien Clochard le mercredi 1 mars 2017 à 21h17

mercredi 22 février 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 19 février 2017

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170219235135.GA1213@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Robert Haas pushed:

Fujii Masao pushed:

Tom Lane pushed:

Magnus Hagander pushed:

Correctifs en attente

Michaël Paquier sent in another revision of a patch to fix an issue where 2PC files could be lost that stemmed from the fact that unlink() is not guaranteed to be durable.

Corey Huinker sent in two more revisions of a patch to add \if and friends to psql.

Ashutosh Sharma sent in a patch to scan has indexes a page at a time, where the previous scans did tuple-at-a-time.

Amit Langote sent in a patch to add relkind checks to certain contrib modules.

Seki Eiji sent in two revisions of a patch to add a GetOldestXmin option which ignores arbitrary vacuum flags.

Thomas Munro sent in another revision of a patch to implement [[Parallel] Shared] Hash.

Amit Khandekar sent in a patch to enable UPDATEs on partitioned tables that would cause a tuple to move from one partition to another.

Vaishnavi Prabakaran and Aya Iwata traded patches to add batch/pipelining support for libpq.

Amit Kapila sent in three more revisions of a patch to add WAL support for hash indexes.

Alexander Korotkov sent in another revision of a patch to cacheline-align PGXACT.

Kyotaro HORIGUCHI and Peter Eisentraut traded patches to make it possible for logical encoding to do the right thing when the encodings of the origin and replica don't match.

Pavel Stěhule sent in another revision of a patch to make it possible to set a template database for pg_regress.

Dilip Kumar sent in three more revisions of a patch to implement parallel bitmap heap scan.

Michaël Paquier sent in a patch to fix some issues with the WAL consistency check facility.

Amit Langote and Ashutosh Bapat traded patches to improve the documentation for partitioning.

Erik Rijkers sent in a patch to fix the docs for CREATE SUBSCRIPTION.

Kyotaro HORIGUCHI sent in another revision of a patch to refactor tab completion in psql.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Tom Lane sent in another revision of a patch to improve OR conditions on joined columns.

Haribabu Kommi sent in another revision of a patch to add a pg_stat_wal_write statistics view.

Masahiko Sawada sent in another revision of a patch to implement transactions involving multiple postgres foreign servers.

Jeff Janes and Neha Khatri traded patches to fix an infelicity between bytea_output and make installcheck.

Michaël Paquier sent in another revision of a patch to fix an issue in PQsendQuery where an error occurs when target_session_attrs is set to read-write.

Masahiko Sawada sent in two more revisions of a patch to fix an infelicity between DROP SUBSCRIPTION and ROLLBACK.

Robert Haas and Amit Kapila traded patches to implement parallel index scans.

Thomas Munro sent in two more revisions of a patch to help measure replay lag.

Kuntal Ghosh sent in a patch to add infrastructure required to expose non-backend processes in pg_stat_activity, use same to expose stats for auxiliary processes in pg_stat_get_activity, expose stats for autovacuum launcher and bgworker, and add a proc_type column in pg_stat_get_activity.

Peter Eisentraut sent in a patch to add a max_worker_processes_per_user setting.

Peter Eisentraut sent in a patch to add errcontext to background worker registration, and hange failures in RegisterBackgroundWorker() to hard errors.

David Christensen sent in a patch to fix some DESCR() grammar mistakes introduced by the xlog -> wal changes.

Anastasia Lubennikova sent in another revision of a patch to add an IF NOT EXISTS option for CREATE SERVER and CREATE USER MAPPING statements.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Thomas Munro sent in another revision of a patch to make it possible to run parallel queries in SERIALIZABLE isolation mode.

Peter Eisentraut sent in another revision of a patch to implement ICU integration.

Haribabu Kommi sent in a patch to allow parallel writers by separating concerns so that the backend does the write operations, while the workers produce the results.

Masahiko Sawada sent in another revision of a patch to add a GUC for cleanup indexes threshold.

David Christensen sent in a patch to teach Catalog.pm how many attributes there should be per DATA() line.

Heikki Linnakangas sent in another revision of a patch to implement SCRAM auth.

Karl O. Pinc and Gilles Darold traded patches to implement pg_current_logfile().

Anastasia Lubennikova sent in another revision of a patch to implement covering and unique indexes.

Amit Kapila sent in two revisions of a patch to ensure that ON CONFLICT DO NOTHING work with partitioned tables.

Rafia Sabih sent in three more revisions of a patch to implement parallel index-only scan.

Tom Lane sent in two revisions of a patch to avoiding OOM in a hash join with many duplicate inner keys.

Rushabh Lathia sent in two more revisions of a patch to implement gather merge.

Rafia Sabih sent in another revision of a patch to allow query string to workers.

Surafel Temsgen sent in another revision of a patch to implement CORRESPONDING.

David Christensen sent in two revisions of a patch to add pg_disable_checksums() and supporting infrastructure.

KaiGai Kohei sent in another revision of a patch to implement a ParallelFinish-hook for FDW/CSP.

Amit Langote sent in a patch to fix the fact that pg_dump would emit ALTER TABLE ONLY for partitioned tables.

Mithun Cy sent in a PoC patch to use a better way to expand hash indexes.

Peter Eisentraut sent in a patch to implement DCL for logical replication.

Amit Khandekar sent in another revision of a patch to implement parallel append.

Michaël Paquier sent in a patch to implement SASLprep aka NFKC for SCRAM authentication.

Tom Lane sent in a patch to add an InterTimestamp struct to fix an issue with floating point timestamps in logical replication.

Alexander Korotkov sent in another revision of a patch to implement incremental sort.

Jim Nasby sent in a patch to deal with the fact that pg_get_object_address() doesn't support composites.

Robert Haas sent in a patch to fix an instability in the select_parallel regression test by ensuring that workers get forgotten faster.

Robins Tharakan sent in a patch to add a --no-pgauthid to pg_dumpall, which will make it at least not fail immediately on Amazon RDS Postgres and similar systems.

Erik Rijkers sent in a patch to fix some comments in origin.c and snapbuild.c

Neha Khatri sent in a patch to fix a typo in varlena.c.

par chl le mercredi 22 février 2017 à 00h43

Nouvelles hebdomadaires de PostgreSQL - 12 février 2017

Versions correctives 9.6.2, 9.5.6, 9.4.11, 9.3.16 et 9.2.20 disponibles. Mettez à jour à la prochaine opportunité d'immobilisation. https://www.postgresql.org/about/news/1733/

Le PGDay suisse se tiendra à Rapperwil le 30 juin 2017. L'appel à conférenciers court jusqu'au 14 avril : http://www.pgday.ch/2017/

[ndt: Meetup à Paris le 9 mars :https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Paris/]

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170212221755.GA21298@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Heikki Linnakangas pushed:

Peter Eisentraut pushed:

  • Add missing newline to error messages. Also improve the message style a bit while we're here. http://git.postgresql.org/pg/commitdiff/afcb0c97efc58459bcbbe795f42d8b7be414e076
  • doc: Update CREATE DATABASE examples. The example of using CREATE DATABASE with the ENCODING option did not work anymore (except in special circumstances) and did not represent a good general-purpose example, so write some new examples. Reported-by: marc+pgsql@milestonerdl.com http://git.postgresql.org/pg/commitdiff/549f74733f45804cd3180de853e5d0610eecc22f
  • doc: Document sequence function privileges better. Document the privileges required for each of the sequence functions. This was already in the GRANT reference page, but also add it to the function description for easier reference. http://git.postgresql.org/pg/commitdiff/696af9ab0a92642978169c227e187a65c2f35f17
  • Avoid permission failure in pg_sequences.last_value. Before, reading pg_sequences.last_value would fail unless the user had appropriate sequence permissions, which would make the pg_sequences view cumbersome to use. Instead, return null instead of the real value when there are no permissions. From: Michael Paquier <michael.paquier@gmail.com> Reported-by: Shinoda, Noriyoshi <noriyoshi.shinoda@hpe.com> http://git.postgresql.org/pg/commitdiff/ab82340a43bebe57a3db0e52bb74120b3bb53ae5
  • doc: Some improvements in CREATE SUBSCRIPTION ref page. Add link to description of libpq connection strings. Add link to explanation of replication access control. This currently points to the description of streaming replication access control, which is currently the same as for logical replication, but that might be refined later. Also remove plain-text passwords from the examples, to not encourage that dubious practice. based on suggestions from Simon Riggs http://git.postgresql.org/pg/commitdiff/e35bbea7ddd89c7f51988fcf03c87150938ea2e3
  • Fix relcache leaks in get_object_address_publication_rel(). http://git.postgresql.org/pg/commitdiff/115cb31597fac8a17202d1e41da8baf33fcb60cf
  • Add CREATE SEQUENCE AS <data type> clause. This stores a data type, required to be an integer type, with the sequence. The sequences min and max values default to the range supported by the type, and they cannot be set to values exceeding that range. The internal implementation of the sequence is not affected. Change the serial types to create sequences of the appropriate type. This makes sure that the min and max values of the sequence for a serial column match the range of values supported by the table column. So the sequence can no longer overflow the table column. This also makes monitoring for sequence exhaustion/wraparound easier, which currently requires various contortions to cross-reference the sequences with the table columns they are used with. This commit also effectively reverts the pg_sequence column reordering in f3b421da5f4addc95812b9db05a24972b8fd9739, because the new seqtypid column allows us to fill the hole in the struct and create a more natural overall column ordering. Reviewed-by: Steve Singer <steve@ssinger.info> Reviewed-by: Michael Paquier <michael.paquier@gmail.com> http://git.postgresql.org/pg/commitdiff/2ea5b06c7a7056dca0af1610aadebe608fbcca08

Tom Lane pushed:

  • Update comment in relcache.c. Commit 665d1fad9 introduced rd_pkindex, and made RelationGetIndexList responsible for updating it, but didn't bother to fix RelationGetIndexList's header comment to say so. http://git.postgresql.org/pg/commitdiff/a59318346ef476d3e66c4d13e92cf4ad6ce328ac
  • Avoid returning stale attribute bitmaps in RelationGetIndexAttrBitmap(). The problem with the original coding here is that we might receive (and clear) a relcache invalidation signal for the target relation down inside one of the index_open calls we're doing. Since the target is open, we would not drop the relcache entry, just reset its rd_indexvalid and rd_indexlist fields. But RelationGetIndexAttrBitmap() kept going, and would eventually cache and return potentially-obsolete attribute bitmaps. The case where this matters is where the inval signal was from a CREATE INDEX CONCURRENTLY telling us about a new index on a formerly-unindexed column. (In all other cases, the lock we hold on the target rel should prevent any concurrent change in index state.) Even just returning the stale attribute bitmap is not such a problem, because it shouldn't matter during the transaction in which we receive the signal. What hurts is caching the stale data, because it can survive into later transactions, breaking CREATE INDEX CONCURRENTLY's expectation that later transactions will not create new broken HOT chains. The upshot is that there's a window for building corrupted indexes during CREATE INDEX CONCURRENTLY. This patch fixes the problem by rechecking that the set of index OIDs is still the same at the end of RelationGetIndexAttrBitmap() as it was at the start. If not, we loop back and try again. That's a little more than is strictly necessary to fix the bug --- in principle, we could return the stale data but not cache it --- but it seems like a bad idea on general principles for relcache to return data it knows is stale. There might be more hazards of the same ilk, or there might be a better way to fix this one, but this patch definitely improves matters and seems unlikely to make anything worse. So let's push it into today's releases even as we continue to study the problem. Pavan Deolasee and myself Discussion: https://postgr.es/m/CABOikdM2MUq9cyZJi1KyLmmkCereyGp5JQ4fuwKoyKEde_mzkQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/2aaec65464dad497536199dea13612b9232eaa3e
  • Release note updates. Add item for last-minute CREATE INDEX CONCURRENTLY fix. Repair a couple of misspellings of patch authors' names. Back-branch updates will follow shortly, but I thought I'd commit this separately just to make it more visible. http://git.postgresql.org/pg/commitdiff/ad6af3fc4256c0e1eecf5d93d510da4b44e0d480
  • Correct thinko in last-minute release note item. The CREATE INDEX CONCURRENTLY bug can only be triggered by row updates, not inserts, since the problem would arise from an update incorrectly being made HOT. Noted by Alvaro. http://git.postgresql.org/pg/commitdiff/39c3ca5161911cebafbcce160ef89e5a4faaf569
  • Speed up "brin" regression test a little bit. In the large DO block, collect row TIDs into array variables instead of creating and dropping a pile of temporary tables. In a normal build, this reduces the brin test script's runtime from about 1.1 sec to 0.4 sec on my workstation. That's not all that exciting perhaps, but in a CLOBBER_CACHE_ALWAYS test build, the runtime drops from 20 min to 17 min, which is a little more useful. In combination with some other changes I plan to propose, this will help provide a noticeable reduction in cycle time for CLOBBER_CACHE_ALWAYS buildfarm critters. http://git.postgresql.org/pg/commitdiff/242066cc8e587ccbe5e1cf38c4f085f080dcd5e0
  • Fix roundoff problems in float8_timestamptz() and make_interval(). When converting a float value to integer microseconds, we should be careful to round the value to the nearest integer, typically with rint(); simply assigning to an int64 variable will truncate, causing apparently off-by-one values in cases that should work. Most places in the datetime code got this right, but not these two. float8_timestamptz() is new as of commit e511d878f (9.6). Previous versions effectively depended on interval_mul() to do roundoff correctly, which it does, so this fixes an accuracy regression in 9.6. The problem in make_interval() dates to its introduction in 9.4. Aside from being careful to round not truncate, let's incorporate the hours and minutes inputs into the result with exact integer arithmetic, rather than risk introducing roundoff error where there need not have been any. float8_timestamptz() problem reported by Erik Nordström, though this is not his proposed patch. make_interval() problem found by me. Discussion: https://postgr.es/m/CAHuQZDS76jTYk3LydPbKpNfw9KbACmD=49dC4BrzHcfPv6yA1A@mail.gmail.com http://git.postgresql.org/pg/commitdiff/8f93bd8512466c9b6c4dbc1e5efd0f72b8e2be9a
  • Allow index AMs to cache data across aminsert calls within a SQL command. It's always been possible for index AMs to cache data across successive amgettuple calls within a single SQL command: the IndexScanDesc.opaque field is meant for precisely that. However, no comparable facility exists for amortizing setup work across successive aminsert calls. This patch adds such a feature and teaches GIN, GIST, and BRIN to use it to amortize catalog lookups they'd previously been doing on every call. (The other standard index AMs keep everything they need in the relcache, so there's little to improve there.) For GIN, the overall improvement in a statement that inserts many rows can be as much as 10%, though it seems a bit less for the other two. In addition, this makes a really significant difference in runtime for CLOBBER_CACHE_ALWAYS tests, since in those builds the repeated catalog lookups are vastly more expensive. The reason this has been hard up to now is that the aminsert function is not passed any useful place to cache per-statement data. What I chose to do is to add suitable fields to struct IndexInfo and pass that to aminsert. That's not widening the index AM API very much because IndexInfo is already within the ken of ambuild; in fact, by passing the same info to aminsert as to ambuild, this is really removing an inconsistency in the AM API. Discussion: https://postgr.es/m/27568.1486508680@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/86d911ec0f9d4643e9a47db42510959dec0ed76b
  • Blind try to fix portability issue in commit 8f93bd851 et al. The S/390 members of the buildfarm are showing failures indicating that they're having trouble with the rint() calls I added yesterday. There's no good reason for that, and I wonder if it is a compiler bug similar to the one we worked around in d9476b838. Try to fix it using the same method as before, namely to store the result of rint() back into a "double" variable rather than immediately converting to int64. (This isn't entirely waving a dead chicken, since on machines with wider-than-double float registers, the extra store forces a width conversion. I don't know if S/390 is like that, but it seems worth trying.) In passing, merge duplicate ereport() calls in float8_timestamptz(). Per buildfarm. http://git.postgresql.org/pg/commitdiff/5d2adf0f81a2e4ca4f101b19b1efea147b462301

Robert Haas pushed:

  • Cache hash index's metapage in rel->rd_amcache. This avoids a very significant amount of buffer manager traffic and contention when scanning hash indexes, because it's no longer necessary to lock and pin the metapage for every scan. We do need some way of figuring out when the cache is too stale to use any more, so that when we lock the primary bucket page to which the cached metapage points us, we can tell whether a split has occurred since we cached the metapage data. To do that, we use the hash_prevblkno field in the primary bucket page, which would otherwise always be set to InvalidBuffer. This patch contains code so that it will continue working (although less efficiently) with hash indexes built before this change, but perhaps we should consider bumping the hash version and ripping out the compatibility code. That decision can be made later, though. Mithun Cy, reviewed by Jesper Pedersen, Amit Kapila, and by me. Before committing, I made a number of cosmetic changes to the last posted version of the patch, adjusted _hash_getcachedmetap to be more careful about order of operation, and made some necessary updates to the pageinspect documentation and regression tests. http://git.postgresql.org/pg/commitdiff/293e24e507838733aba4748b514536af2d39d7f2
  • Fix compiler warning. Mithun Cy, per a report by Erik Rijkers http://git.postgresql.org/pg/commitdiff/94708c0e8c32ad1c9c6ffbdb894fe158eda596e7
  • Allow the element allocator for a simplehash to be specified. This is infrastructure for a pending patch to allow parallel bitmap heap scans. Dilip Kumar, reviewed (in earlier versions) by Andres Freund and (more recently) by me. Some further renaming by me, also. http://git.postgresql.org/pg/commitdiff/565903af474e85cef28ff712d773f13b6701ded5
  • Avoid redefining simplehash_allocate/simplehash_free. There's no generic guard against multiple inclusion in this file, for good reason. But these typedefs need one, as per a report from Jeff Janes. http://git.postgresql.org/pg/commitdiff/ac8eb972f268c787bfe7579b1747c3866fced5a9
  • Revise the way the element allocator for a simplehash is specified. This method is more elegant and more efficient. Per a suggestion from Andres Freund, who also briefly reviewed the patch. http://git.postgresql.org/pg/commitdiff/c3c4f6e1740be256178cd7551d5b8a7677159b74
  • Add WAL consistency checking facility. When the new GUC wal_consistency_checking is set to a non-empty value, it triggers recording of additional full-page images, which are compared on the standby against the results of applying the WAL record (without regard to those full-page images). Allowable differences such as hints are masked out, and the resulting pages are compared; any difference results in a FATAL error on the standby. Kuntal Ghosh, based on earlier patches by Michael Paquier and Heikki Linnakangas. Extensively reviewed and revised by Michael Paquier and by me, with additional reviews and comments from Amit Kapila, Ãlvaro Herrera, Simon Riggs, and Peter Eisentraut. http://git.postgresql.org/pg/commitdiff/a507b86900f695aacc8d52b7d2cfcb65f58862a2
  • pageinspect: Fix hash_bitmap_info not to read the underlying page. It did that to verify that the page was an overflow page rather than anything else, but that means that checking the status of all the overflow bits requires reading the entire index. So don't do that. The new code validates that the page is not a primary bucket page or bitmap page by looking at the metapage, so that using this on large numbers of pages can be reasonably efficient. Ashutosh Sharma, per a complaint from me, and with further modifications by me. http://git.postgresql.org/pg/commitdiff/fc8219dc54c95ea673560b786aa8123ce6ec5977
  • Fix race condition in ConditionVariablePrepareToSleep. Thomas Munro http://git.postgresql.org/pg/commitdiff/3f3d60d3bbd10f6cc118d24aadc60e96b9854576
  • simplehash: Additional tweaks to make specifying an allocator work. Even if we don't emit definitions for SH_ALLOCATE and SH_FREE, we still need prototypes. The user can't define them before including simplehash.h because SH_TYPE isn't available yet. For the allocator to be able to access private_data, it needs to become an argument to SH_CREATE. Previously we relied on callers to set that after returning from SH_CREATE, but SH_CREATE calls SH_ALLOCATE before returning. Dilip Kumar, reviewed by me. http://git.postgresql.org/pg/commitdiff/72257f95781af97108fa9a9e7224ec81a90e7693
  • Remove all references to "xlog" from SQL-callable functions in pg_proc. Commit f82ec32ac30ae7e3ec7c84067192535b2ff8ec0e renamed the pg_xlog directory to pg_wal. To make things consistent, and because "xlog" is terrible terminology for either "transaction log" or "write-ahead log" rename all SQL-callable functions that contain "xlog" in the name to instead contain "wal". (Note that this may pose an upgrade hazard for some users.) Similarly, rename the xlog_position argument of the functions that create slots to be called wal_position. Discussion: https://www.postgresql.org/message-id/CA+Tgmob=YmA=H3DbW1YuOXnFVgBheRmyDkWcD9M8f=5bGWYEoQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/806091c96f9b81f7631e4e37a05af377b473b5da
  • Rename user-facing tools with "xlog" in the name to say "wal". This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump. http://git.postgresql.org/pg/commitdiff/85c11324cabaddcfaf3347df78555b30d27c5b5a
  • Rename dtrace probes for ongoing xlog -> wal conversion. xlog-switch becomes wal-switch, and xlog-insert becomes wal-insert. http://git.postgresql.org/pg/commitdiff/3f01fd4ca0b4c81333b1f0dadb09c73aa589ab6e
  • Rename command line options for ongoing xlog -> wal conversion. initdb and pg_basebackup now have a --waldir option rather --xlogdir, and pg_basebackup now has --wal-method rather than --xlog-method. http://git.postgresql.org/pg/commitdiff/62e8b387514ce965c8b3d67c81990e0ecf8c9b83

Andres Freund pushed:

Simon Riggs pushed:

Noah Misch pushed:

  • Ignore tablespace ACLs when ignoring schema ACLs. The ALTER TABLE ALTER TYPE implementation can issue DROP INDEX and CREATE INDEX to refit existing indexes for the new column type. Since this CREATE INDEX is an implementation detail of an index alteration, the ensuing DefineIndex() should skip ACL checks specific to index creation. It already skips the namespace ACL check. Make it skip the tablespace ACL check, too. Back-patch to 9.2 (all supported versions). Reviewed by Tom Lane. http://git.postgresql.org/pg/commitdiff/f30f34e5897b64e0fb6616154c11dc9765866046

Correctifs en attente

Amit Khandekar sent in two more revisions of a patch to implement Parallel Append.

Boris Muratshin sent in a patch to implement 3D Z-curve spatial indexes.

Corey Huinker sent in five more revisions of a patch to implement \if and friends in psql.

Nikita Glukhov sent in another revision of a patch to implement KNN for B-trees.

Nikita Glukhov sent in another revision of a patch to implement KNN for SP-GiST.

Beena Emerson sent in another revision of a patch to allow increasing the default WAL segment size.

Nikolay Shaplov sent in another revision of a patch to move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM.

Fujii Masao sent in a patch to fix a bug that made it impossible to shut down a subscriber after DROP SUBSCRIPTION.

Heikki Linnakangas sent in another revision of a patch to implement SCRAM authentication.

Christoph Berg sent in two more revisions of a patch to implement \gx, a one-shot expanded output for queries, in psql.

Mithun Cy sent in two more revisions of a patch to implement auto_prewarm.

Rafia Sabih sent in two more revisions of a patch to enable passing query string to workers.

Takeshi Ideriha sent in another revision of a patch to enable DECLARE STATEMENT to set up a connection in ECPG.

Peter Eisentraut sent in a patch to drop Python 2.3 support.

Pavel Raiskup sent in two revisions of a patch to create a configure-time knob to set default ssl ciphers.

Naoki Okano sent in a patch to implement CREATE OR REPLACE TRIGGER.

Piotr Stefaniak sent in a patch to pg_bsd_indent to implement -lps ("leave preprocessor space").

Dilip Kumar and Robert Haas traded patches to implement parallel bitmap heap scan.

Masahiko Sawada sent in two revisions of a patch to stop the apply worker after DROP SUBSCRIPTION is committed.

Peter Eisentraut sent in a patch to systematically trim the trailing newlines off PQerrorMessage() results in backend uses (dblink, postgres_fdw, libpqwalreceiver).

Peter Eisentraut sent in a patch to implement CREATE COLLATION IF NOT EXISTS.

Michaël Paquier sent in a patch to implement SASLprep(), or NFKC if you want for UTF-8 strings.

Kyle Gearhart sent in a patch to implement an alternate row processor for libpq which is faster for certain use cases than the default one.

Amit Langote sent in two revisions of a patch to implement a check partition strategy in ATExecDropNotNull.

Andres Freund sent in a patch to speed up expression processing, including several JIT PoCs.

Pavel Stěhule sent in two revisions of a patch to enable specifying a template database for pg_regress.

Petr Jelínek sent in another revision of a patch to enable existing data copy for logical replication.

Amit Kapila sent in two more revisions of a patch to implement parallel index scans.

Amit Kapila sent in another revision of a patch to implement WAL for hash indexes.

Amit Langote sent in a patch to optimize partitioned tables by noting that top-level tables are always empty and avoiding that anything that might write to them can't.

Amit Langote sent in a patch to add relkind checks to certain contrib modules.

Peter Geoghegan sent in a patch to add parallel B-tree index build sorting with some testing tools.

Ashutosh Bapat sent in three more revisions of a patch to speed up partition-wise joins on declaratively partitioned tables.

Amit Kapila sent in another revision of a patch to parallelize queries containing initplans.

Simon Riggs sent in a patch to make log_autovacuum_min_duration log the durations of vacuums whether or not they were launched by autovacuum workers.

Simon Riggs sent in a patch to enable reporting xmin from VACUUMs.

Peter Geoghegan sent in a patch to add amcheck extension to contrib.

Michael Banck sent in three revisions of a patch to better document pg_basebackup's behavior in certain corner cases.

Andreas Karlsson sent in another revision of a patch to implement REINDEX CONCURRENTLY.

Tom Lane sent in a patch to preprocess join OR clauses that might be better handled as UNIONs.

Magnus Hagander sent in a patch to enable having fallback servers RADIUS auth.

par chl le mercredi 22 février 2017 à 00h39

jeudi 9 février 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 5 février 2017

Le premier pgDay Israel 2017 aura lieu le 2 mars 2017. Les inscriptions sont ouvertes : http://pgday.org.il/

Postgres Vision aura lieu à Boston du 26 au 28 juin 2017. L'appel à conférenciers court jusqu'au 24 février 2017 : écrire à CFP AT PostgresVision DOT com : http://postgresvision.com/

[ndt: Meetup à Nantes le 8 mars :https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Nantes/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170205221731.GA32585@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Stephen Frost pushed:

  • Handle ALTER EXTENSION ADD/DROP with pg_init_privs. In commit 6c268df, pg_init_privs was added to track the initial privileges of catalog objects and extensions. Unfortunately, that commit didn't include understanding of ALTER EXTENSION ADD/DROP, which allows the objects associated with an extension to be changed after the initial CREATE EXTENSION script has been run. The result of this meant that ACLs for objects added through ALTER EXTENSION ADD were not recorded into pg_init_privs and we would end up including those ACLs in pg_dump when we shouldn't have. This commit corrects that by making sure to have pg_init_privs updated when ALTER EXTENSION ADD/DROP is run, recording the permissions as they are at ALTER EXTENSION ADD time, and removing any if/when ALTER EXTENSION DROP is called. This issue was pointed out by Moshe Jacobson as commentary on bug #14456 (which was actually a bug about versions prior to 9.6 not handling custom ACLs on extensions correctly, an issue now addressed with pg_init_privs in 9.6). Back-patch to 9.6 where pg_init_privs was introduced. http://git.postgresql.org/pg/commitdiff/e54f75722c720b596ec5e72154cc899da199de5b
  • test_pg_dump TAP test whitespace cleanup. The formatting of the perl hashes used in the TAP tests for test_pg_dump was rather horribly inconsistent and made it more difficult than it really should have been to add new tests or adjust what tests are for what runs, etc. Reformat to clean that all up. Whitespace-only changes. http://git.postgresql.org/pg/commitdiff/fb94ca77f1400e236b00d432dccfbe4f1124971c
  • test_pg_dump: perltidy cleanup. As pointed out by Alvaro, we actually use perltidy on the perl scripts in the source tree, so go back to the results of a perltidy run for the test_pg_dump TAP script. To make it look slightly less tragic, I changed most of the independent arguments into long-form single arguments (eg: -f file.sql changed to be --file=file.sql) to avoid having them confusingly split across lines due to perltidy. Back-patch to 9.6, as the last patch was. http://git.postgresql.org/pg/commitdiff/58da8334308c26107ebb7ee06c99589e14bd882b
  • perltidy pg_dump TAP tests. The pg_dump TAP tests have gotten pretty far from what perltidy thinks they should be, so fix that, and in passing use long-form argument names with arguments passed via "=" in a similar vein to 58da833. No functional changes here, just whitespace and changing runs from "-f" to "--file=", and similar. http://git.postgresql.org/pg/commitdiff/6af8b89adba16f97bee0d3b01256861e10d0e4f1
  • pg_dump: Fix handling of ALTER DEFAULT PRIVILEGES. In commit 23f34fa, we changed how ACLs were handled to use the new pg_init_privs catalog and to dump out the ACL commands as REVOKE+GRANT combinations instead of trying to REVOKE all rights always and then GRANT back just the ones which were in place. Unfortunately, the DEFAULT PRIVILEGES system didn't quite get the correct treatment with this change and ended up (incorrectly) only including positive GRANTs instead of both the REVOKEs and GRANTs necessary to preserve the correct privileges. There are only a couple cases where such REVOKEs are possible because, generally speaking, there's few rights which exist on objects by default to be revoked. Examples of REVOKEs which weren't being correctly preserved are when privileges are REVOKE'd from the creator/owner, like so: ALTER DEFAULT PRIVILEGES FOR ROLE myrole REVOKE SELECT ON TABLES FROM myrole; or when other default privileges are being revoked, such as EXECUTE rights granted to public for functions: ALTER DEFAULT PRIVILEGES FOR ROLE myrole REVOKE EXECUTE ON FUNCTIONS FROM PUBLIC; Fix this by correctly working out what the correct REVOKE statements are (if any) and dump them out, just as we do for everything else. Noticed while developing additional regression tests for pg_dump, which will be landing shortly. Back-patch to 9.6 where the bug was introduced. http://git.postgresql.org/pg/commitdiff/e2090d9d20d8091c9478a674d9c22fc8006909ce

Heikki Linnakangas pushed:

Tom Lane pushed:

  • Update time zone data files to tzdata release 2016j. DST law changes in northern Cyprus (new zone Asia/Famagusta), Russia (new zone Europe/Saratov), Tonga, Antarctica/Casey. Historical corrections for Asia/Aqtau, Asia/Atyrau, Asia/Gaza, Asia/Hebron, Italy, Malta. Replace invented zone abbreviation "TOT" for Tonga with numeric UTC offset; but as in the past, we'll keep accepting "TOT" for input. http://git.postgresql.org/pg/commitdiff/308d8682740391cf6e94fdbadcdbbecfa02ea208
  • Make psql reject attempts to set special variables to invalid values. Previously, if the user set a special variable such as ECHO to an unrecognized value, psql would bleat but store the new value anyway, and then fall back to a default setting for the behavior controlled by the variable. This was agreed to be a not particularly good idea. With this patch, invalid values result in an error message and no change in state. (But this applies only to variables that affect psql's behavior; purely informational variables such as ENCODING can still be set to random values.) To do this, modify the API for psql's assign-hook functions so that they can return an OK/not OK result, and give them the responsibility for printing error messages when they reject a value. Adjust the APIs for ParseVariableBool and ParseVariableNum to support the new behavior conveniently. In passing, document the variable VERSION, which had somehow escaped that. And improve the quite-inadequate commenting in psql/variables.c. Daniel Vérité, reviewed by Rahila Syed, some further tweaking by me Discussion: https://postgr.es/m/7356e741-fa59-4146-a8eb-cf95fd6b21fb@mm http://git.postgresql.org/pg/commitdiff/511ae628f31b4e791cd5c7836e46cb84dcf145fd
  • Add a regression test script dedicated to exercising system views. Quite a few of our built-in system views were not exercised anywhere in the regression tests. This is perhaps not so exciting for the ones that are simple projections/joins of system catalogs, but for the ones that are wrappers for set-returning C functions, the omission translates directly to lack of test coverage for those functions. In many cases, the reason for the omission is that the view doesn't have much to do with any specific SQL feature, so there's no natural place to test it. To remedy that, invent a new script sysviews.sql that's dedicated to testing SRF-based views. Move a couple of tests that did fit this charter into the new script, and add simple "count(*)" based tests of other views within the charter. That's enough to ensure we at least exercise the main code path through the SRF, although it does little to prove that the output is sane. More could be done here, no doubt, and I hope someone will think about how we can test these views more thoroughly. But this is a starting point. Discussion: https://postgr.es/m/19359.1485723741@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/d002f16c6ec38f76d1ee97367ba6af3000d441d0
  • Invent pg_hba_file_rules view to show the content of pg_hba.conf. This view is designed along the same lines as pg_file_settings, to wit it shows what is currently in the file, not what the postmaster has loaded as the active settings. That allows it to be used to pre-vet edits before issuing SIGHUP. As with the earlier view, go out of our way to allow errors in the file to be reflected in the view, to assist that use-case. (We might at some point invent a view to show the current active settings, but this is not that patch; and it's not trivial to do.) Haribabu Kommi, reviewed by Ashutosh Bapat, Michael Paquier, Simon Riggs, and myself Discussion: https://postgr.es/m/CAJrrPGerH4jiwpcXT1-46QXUDmNp2QDrG9+-Tek_xC8APHShYw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/de16ab7238888b16825ad13f0bbe123632915e9b
  • Simplify some long-obsolete code in hba.c's next_token(). next_token() oddly set its buffer space consumption limit to one before the last char position in the buffer, not the last as you'd expect. The reason is there was once an ugly kluge to mark keywords by appending a newline to them, potentially requiring one more byte. Commit e5e2fc842 removed that kluge, but failed to notice that the length limit could be increased. Also, remove some vestigial handling of newline characters in the buffer. That was left over from when this function read the file directly using getc(). Commit 7f49a67f9 changed it to read from a buffer, from which tokenize_file had already removed the only possible occurrence of newline, but did not simplify this function in consequence. Also, ensure that we don't return with *lineptr set to someplace past the terminating '\0'; that would be catastrophic if a caller were to ask for another token from the same line. This is just latent since no callers actually do call again after a "false" return; but considering that it was actually costing us extra code to do it wrong, we might as well make it bulletproof. Noted while reviewing pg_hba_file_rules patch. http://git.postgresql.org/pg/commitdiff/1e5a5d03da14ba9b734c6a2a1a822dcd8af110eb
  • Make psql's \set display variables in alphabetical order. "\set" with no arguments displays all defined variables, but it does so in the order that they appear in variables.c's list, which previously was mostly creation order. That makes the list ugly and hard to find things in, and it exposes some psql implementation details to users. (For instance, ordinary variables will move to the bottom of the list if unset and set again, but variables that have hooks won't.) Fix that by keeping the list in alphabetical order at all times, which isn't much more complicated than breaking out of the insertion search loops once we reach an entry that should be after the one to be inserted. Discussion: https://postgr.es/m/31785.1485900786@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c3e3844a92fe42109e4106314f7d229f784a7d0a
  • Improve psql's behavior for \set and \unset of its control variables. This commit improves on the results of commit 511ae628f in two ways: 1. It restores the historical behavior that "\set FOO" is interpreted as setting FOO to "on", if FOO is a boolean control variable. We already found one test script that was expecting that behavior, and the psql documentation certainly does nothing to discourage people from assuming that would work, since it often says just "if FOO is set" when describing the effects of a boolean variable. However, now this case will result in actually setting FOO to "on", not an empty string. 2. It arranges for an "\unset" of a control variable to set the value back to its default value, rather than becoming apparently undefined. The control variables are also initialized that way at psql startup. In combination, these things guarantee that a control variable always has a displayable value that reflects what psql is actually doing. That is a pretty substantial usability improvement. The implementation involves adding a second type of variable hook function that is able to replace a proposed new value (including NULL) with another one. We could alternatively have complicated the API of the assign hook, but this way seems better since many variables can share the same substitution hook function. Also document the actual behavior of these variables more fully, including covering assorted behaviors that were there before but never documented. This patch also includes some minor cleanup that should have been in 511ae628f but was missed. Patch by me, but it owes a lot to discussions with Daniel Vérité. Discussion: https://postgr.es/m/9572.1485821620@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/86322dc7e013b4062393dcbb74043db003e23ec5
  • Provide CatalogTupleDelete() as a wrapper around simple_heap_delete(). This extends the work done in commit 2f5c9d9c9 to provide a more nearly complete abstraction layer hiding the details of index updating for catalog changes. That commit only invented abstractions for catalog inserts and updates, leaving nearby code for catalog deletes still calling the heap-level routines directly. That seems rather ugly from here, and it does little to help if we ever want to shift to a storage system in which indexing work is needed at delete time. Hence, create a wrapper function CatalogTupleDelete(), and replace calls of simple_heap_delete() on catalog tuples with it. There are now very few direct calls of [simple_]heap_delete remaining in the tree. Discussion: https://postgr.es/m/462.1485902736@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/ab02896510e26e46b830c87eef2c03dd3c52c09e
  • Fix CatalogTupleInsert/Update abstraction for case of shared indstate. Add CatalogTupleInsertWithInfo and CatalogTupleUpdateWithInfo to let callers use the CatalogTupleXXX abstraction layer even in cases where we want to share the results of CatalogOpenIndexes across multiple inserts/updates for efficiency. This finishes the job begun in commit 2f5c9d9c9, by allowing some remaining simple_heap_insert/update calls to be replaced. The abstraction layer is now complete enough that we don't have to export CatalogIndexInsert at all anymore. Also, this fixes several places in which 2f5c9d9c9 introduced performance regressions by using retail CatalogTupleInsert or CatalogTupleUpdate even though the previous coding had been able to amortize CatalogOpenIndexes work across multiple tuples. A possible future improvement is to arrange for the indexing.c functions to cache the CatalogIndexState somewhere, maybe in the relcache, in which case we could get rid of CatalogTupleInsertWithInfo and CatalogTupleUpdateWithInfo again. But that's a task for another day. Discussion: https://postgr.es/m/27502.1485981379@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/aedd554f84bb3cedb32e6e2a954a70537a4da6b9
  • Fix mishandling of tSRFs at different nesting levels. Given a targetlist like "srf(x), f(srf(x))", split_pathtarget_at_srfs() decided that it needed two levels of ProjectSet nodes, failing to notice that the two SRF calls are textually equal(). Because of that, setrefs.c would convert the upper ProjectSet's tlist to "Var1, f(Var1)" (where Var1 represents a reference to the srf(x) output of the lower ProjectSet). This triggered an assertion in nodeProjectSet.c complaining that it found no SRFs to evaluate, as reported by Erik Rijkers. What we want in such a case is to evaluate srf(x) only once and use a plain Result node to compute "Var1, f(Var1)"; that gives results similar to what previous versions produced, whereas allowing srf(x) to be evaluated again in an upper ProjectSet would square the number of rows emitted. Furthermore, even if the SRF calls aren't textually identical, we want them to be evaluated in lockstep, because that's what happened in the old implementation. But split_pathtarget_at_srfs() got this completely wrong, using two levels of ProjectSet for a case like "srf(x), f(srf(y))". Hence, rewrite split_pathtarget_at_srfs() from the ground up so that it groups SRFs according to the depth of nesting of SRFs in their arguments. This is pretty much how we envisioned that working originally, but I blew it when it came to implementation. In passing, optimize the case of target == input_target, which I noticed is not only possible but quite common. Discussion: https://postgr.es/m/dcbd2853c05d22088766553d60dc78c6@xs4all.nl http://git.postgresql.org/pg/commitdiff/c82d4e658eff287846fe6243966d110ff4ae9025
  • Fix placement of initPlans when forcibly materializing a subplan. If we forcibly place a Material node atop a finished subplan, we need to move any initPlans attached to the subplan up to the Material node, in order to keep SS_finalize_plan() happy. I'd figured this out in commit 7b67a0a49 for the case of materializing a cursor plan, but out of an abundance of caution, I put the initPlan movement hack at the call site for that case, rather than inside materialize_finished_plan(). That was the wrong thing, because it turns out to also be necessary for the only other caller of materialize_finished_plan(), ie subselect.c. We lacked any test cases that exposed the mistake, but bug#14524 from Wei Congrui shows that it's possible to get an initPlan reference into the top tlist in that case too, and then SS_finalize_plan() complains. Hence, move the hack into materialize_finished_plan(). In HEAD, also relocate some recently-added tests in subselect.sql, which I'd unthinkingly dropped into the middle of a sequence of related tests. Report: https://postgr.es/m/20170202060020.1400.89021@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/555494d1bc119173bbf712940da823303644d4de
  • Avoid improbable null pointer dereference in pgpassfileWarning(). Coverity complained that we might pass a null pointer to strcmp() if PQresultErrorField were to return NULL. That shouldn't be possible, since the server is supposed to always provide some SQLSTATE or other in an error message. But we usually defend against such hazards, and it only takes a little more code to do so here. There's no good reason to think this is a live bug, so no back-patch. http://git.postgresql.org/pg/commitdiff/8ac0365c22f20dcd2a6b6752b95b665ada83e858
  • Clean up psql's behavior for a few more control variables. Modify FETCH_COUNT to always have a defined value, like other control variables, mainly so it will always appear in "\set" output. Add hooks to force HISTSIZE to be defined and require it to have an integer value. (I don't see any point in allowing it to be set to non-integral values.) Add hooks to force IGNOREEOF to be defined and require it to have an integer value. Unlike the other cases, here we're trying to be bug-compatible with a rather bogus externally-defined behavior, so I think we need to continue to allow "\set IGNOREEOF whatever". Fix it so that the substitution hook silently replace non-numeric values with "10", so that the stored value always reflects what we're really doing. Add a dummy assign hook for HISTFILE, just so it's always in variables.c's list. We can't require it to be defined always, because that would break the interaction with the PSQL_HISTORY environment variable, so there isn't any change in visible behavior here. Remove tab-complete.c's private list of known variable names, since that's really a maintenance nuisance. Given the preceding changes, there are no control variables it won't show anyway. This does mean that if for some reason you've unset one of the status variables (DBNAME, HOST, etc), that variable would not appear in tab completion for \set. But I think that's fine, for at least two reasons: we shouldn't be encouraging people to use those variables as regular variables, and if someone does do so anyway, why shouldn't it act just like a regular variable? Remove ugly and no-longer-used-anywhere GetVariableNum(). In general, future additions of integer-valued control variables should follow the paradigm of adding an assign hook using ParseVariableNum(), so there's no reason to expect we'd need this again later. Discussion: https://postgr.es/m/17516.1485973973@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/fd6cd698031d3dbeccafde49a8b118cb840312aa
  • Fix a bunch more portability bugs in commit 08bf6e529. It seems like somebody used a dartboard while choosing integer widths for the various values taken and returned by these functions ... and then threw a fresh set of darts while writing the SQL declarations. This patch brings the C code into line with what the SQL declarations say, which is enough to make it not dump core on the particular 32-bit machine I'm testing on. But I think we could do with another round of looking at what the datum widths *should* be. For instance, it's not all that sensible that hash_bitmap_info decided to use int64 to represent a BlockNumber input when get_raw_page doesn't do it that way. There's also a remaining problem that the expected outputs from the test script are platform-dependent, but I'll leave that issue for somebody else. Per buildfarm. http://git.postgresql.org/pg/commitdiff/c6eeb67dcc3b3f7aa94335e18a1075863d808a7c
  • In pageinspect/hashfuncs.c, avoid crashes on alignment-picky machines. On machines with MAXALIGN = 8, the payload of a bytea is not maxaligned, since it will start 4 bytes into a palloc'd value. On alignment-picky hardware, this will cause failures in accesses to 8-byte-wide values within the page. We already encountered this problem when we introduced GIN index inspection functions, and fixed it in commit 84ad68d64. Make use of the same function for hash indexes. A small difficulty is that up to now contrib/pageinspect has not shared any functions at all across files. To support that, introduce a common header file "pageinspect.h" for the module. Also, move get_page_from_raw() out of ginfuncs.c, where it didn't especially belong, and put it in rawpage.c which seems a more natural home. Per buildfarm. Discussion: https://postgr.es/m/17311.1486134714@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/14e9b18fed289e483ed28daec60fdab95da9cc48
  • First-draft release notes for 9.6.2. As usual, the release notes for other branches will be made by cutting these down, but put them up for community review first. http://git.postgresql.org/pg/commitdiff/9863017b87f3592ff663d03fc663a4f1f8fdb8b2

Peter Eisentraut pushed:

Ãlvaro Herrera pushed:

  • Tweak catalog indexing abstraction for upcoming WARM. Split the existing CatalogUpdateIndexes into two different routines, CatalogTupleInsert and CatalogTupleUpdate, which do both the heap insert/update plus the index update. This removes over 300 lines of boilerplate code all over src/backend/catalog/ and src/backend/commands. The resulting code is much more pleasing to the eye. Also, by encapsulating what happens in detail during an UPDATE, this facilitates the upcoming WARM patch, which is going to add a few more lines to the update case making the boilerplate even more boring. The original CatalogUpdateIndexes is removed; there was only one use left, and since it's just three lines, we can as well expand it in place there. We could keep it, but WARM is going to break all the UPDATE out-of-core callsites anyway, so there seems to be no benefit in doing so. Author: Pavan Deolasee Discussion: https://www.postgr.es/m/CABOikdOcFYSZ4vA2gYfs=M2cdXzXX4qGHeEiW3fu9PCfkHLa2A@mail.gmail.com http://git.postgresql.org/pg/commitdiff/2f5c9d9c9cec436e55847ec580606d7e88067df6

Robert Haas pushed:

Andrew Dunstan pushed:

  • Don't count background workers against a user's connection limit. Doing so doesn't seem to be within the purpose of the per user connection limits, and has particularly unfortunate effects in conjunction with parallel queries. Backpatch to 9.6 where parallel queries were introduced. David Rowley, reviewed by Robert Haas and Albe Laurenz. http://git.postgresql.org/pg/commitdiff/f1169ab501ce90e035a7c6489013a1d4c250ac92

Noah Misch pushed:

  • Code review for avoidance of direct cross-module links. Remove $(pkglibdir) from $(rpathdir), since commits d51924be886c2a05e691fa05b16cb6b30ab8370f and eda04886c1e048d695728206504ab4198462168e removed direct linkage to objects stored there. Users are unlikely to notice the difference. Accompany every $(python_libspec) with $(python_additional_libs); this doesn't fix a demonstrated bug, but it might do so on rare Python configurations. With these changes, AIX ceases to be a special case. http://git.postgresql.org/pg/commitdiff/acd73ad1a17f7aed6a914dd9c331f1043d89988d

Fujii Masao pushed:

Correctifs en attente

Takayuki Tsunakawa sent in another revision of a patch to support huge pages on Windows.

Nikita Glukhov sent in three revisions of a patch to fix a bug in SP-GiST box_ops.

Kyotaro HORIGUCHI sent in two more revisions of a patch to implement character conversion with radix trees.

Alexander Korotkov sent in another revision of a patch to cacheline align PGXACT.

Nikita Glukhov sent in a patch to implement kNN searches for SP-GiST.

Michaël Paquier sent in another revision of a patch to allow interrupts on waiting standby.

Rushabh Lathia sent in a patch to extend the existing wait event infrastructure to implement the wait events for the disk I/O.

Etsuro Fujita sent in another revision of a patch to push down more full joins to the PostgreSQL FDW.

Michaël Paquier sent in two more revisions of a patch to refactor replication commands using printsimple.

David Rowley sent in two more revisions of a patch to for improve performance for outer joins where outer side is unique.

Daisuke Higuchi and Ashutosh Bapat traded patches to fix a bug where PQgetResult is not called until PQgetResult has returned a null pointer.

Heikki Linnakangas sent in two more revisions of a patch to fix an issue which produced a deadlock in XLogInsert on AIX.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous execution.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Amit Kapila sent in two more revisions of a patch to implement parallel index scans.

Claudio Freire sent in two more revisions of a patch to enable VACUUM to use more than 1GB of work_mem.

Pavan Deolasee, Ãlvaro Herrera, and Tom Lane traded patches to fix an issue where CREATE INDEX CONCURRENTLY could cause index corruption.

Amit Kapila sent in another revision of a patch to parallelize queries containing initplans.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Michaël Paquier sent in a patch to forbid newline and carriage return characters in database and role names, and ensure clean-up of data directory even with restricted path applied.

Haribabu Kommi sent in another revision of a patch to add a macaddr 64 bit (EUI-64) datatype.

Peter Eisentraut sent in another revision of a patch to implement a set of SEQUENCE data types.

Pavan Deolasee sent in three more revisions of a patch to implement WARM.

Kyotaro HORIGUCHI sent in two more revisions of a patch to fix a bug in physical replication slots.

Kyotaro HORIGUCHI sent in two more revisions of a patch to enable logical replication between databases with different encodings.

Dilip Kumar sent in two more revisions of a patch to implement parallel bitmap heap scan.

Pavel Stěhule sent in another revision of a patch to enable PL/pgsql to force a custom or generic plan.

Anastasia Lubennikova sent in a patch to allow casting from the appropriate jsonb objects to numeric, int, float, bool.

Dagfinn Ilmari Mannsåker sent in two revisions of a patch to add tab completion for DEALLOCATE.

Rushabh Lathia sent in two more revisions of a patch to implement Gather Merge.

Nikhil Sontakke sent in two more revisions of a patch to speed up two-phase transactions.

David Rowley sent in a patch to fix a compiler warning in dbd69118.

Michaël Paquier sent in two revisions of a patch to provide list of subscriptions and publications in psql's completion.

Corey Huinker sent in three more revisions of a patch to add \if and friends to psql.

Peter Moser sent in another revision of a patch to add temporal query primitives.

Alexey Bashtanov sent in a patch to optimize the query used to produce information_schema.constraint_column_usage.

Simon Riggs sent in a patch to introduce the concept of a "superowner".

Andres Freund sent in a patch to implement a basic lockless single producer, multiple consumer ringbuffer, and a rewrite of the bgwriter to use same.

Kyotaro HORIGUCHI sent in another revision of a patch to refactor tab completion in psql and use same to add some tab completion features.

Amit Langote sent in a patch to improve the documentation about partitioned tables.

Andreas Karlsson sent in a patch to add psql tab completion for \help DROP|ALTER.

Alexander Korotkov sent in a patch to implement low-level locking on PowerPC in assembly.

Boris Muratshin sent in a patch to implement 3D Z-curve spatial indexes.

Fabien COELHO sent in another revision of a patch to add more operators and functions to pgbench.

Alexander Korotkov sent in a patch to re-add strategy to PinBuffer.

Masahiko Sawada sent in a patch to fix a typo in a variable name in launcher.c.

Noah Misch sent in a patch to ignore tablespace ACLs when ignoring schema ACLs.

Brandur Leach sent in another revision of a patch to implement SortSupport for macaddr data type.

par chl le jeudi 9 février 2017 à 00h34

mardi 7 février 2017

Nicolas Gollet

Backends générant des fichiers temporaires

Dans certain cas il est peut être utile d'obtenir en temps réel la liste des backends générant des fichiers temporaire sur disque.

Lors de certaines opérations (trie, hachages...), PostgreSQL écrit dans des fichiers temporaires. Ces fichiers sont placés dans le sous-répertoire pgsql_tmp et sont nommés de la façon suivante :
pgsql_tmpXXXX.YXXXX correspond au PID du processus qui a créé le fichier et Y au numéro de fichier créé. Par exemple voici deux fichiers temporaires : pgsql_tmp90630.8 et pgsql_tmp90630.9. Lorsque vous utilisez des tablespaces ces fichiers temporaires sont stockés dans l'emplacement disque définit par ceux-ci ceux qui rajoute une complexité supplémentaire pour récupéré ces fichiers.

Une fois le PID identifié, vous pouvez obtenir les informations depuis la vue pg_stat_activity afin d'obtenir les informations sur le backend générant des fichiers temporaires :
select * from pg_stat_activity where pid = <PID>;

Afin de simplifier la récupération de ces informations vous pouvez obtenir ces informations de façon automatique en utilisant la requête SQL ci-dessous :

Cette requête adaptée du projet pgstats permet d'obtenir l'ensemble des requêtes en cours d’exécution générant des fichiers temporaires tout en prenant en compte les éventuelles tablespace.

Vous pouvez aussi utiliser le script bash psql_show_tempfiles.bash présent sur mon Gitub.

par Nicolas GOLLET le mardi 7 février 2017 à 15h44

dimanche 5 février 2017

Damien Clochard

PostgreSQL Temboard : Guide de Démarrage

Le prochain défi pour la communauté PostgreSQL est de produire des outils graphiques riches et bien conçus. C’est un domain dans lequel nous avons une grande marge de progression et Temboard est un des projets (parmi d’autres) qui a l’ambition de devenir un outil graphique pour PostgreSQL.

Temboard : PostgreSQL Remote Control

Puisque que nous parlons d’un outil graphique, allons directement jetez un coup d’oeuil ! Voici 3 étapes pour déployer un environement de test composé de 3 instances Postgres et un serveur central Temboard.

1- Installer docker et Installer docker-compose

2- Récuperer fichier docker compose pré-configuré et le lancer

wget https://raw.githubusercontent.com/dalibo/docker/master/temboard/docker-compose.yml
docker-compose up

3- Aller sur https://127.0.0.1:8888

  • Connectez-vous sur le serveur Temboard avec les codes admin / admin
  • Connectez-vous sur chaque instance Postgres avec alice / alice

Vous devriez voir apparaitre quelque chose comme ça :

Temboard PostgreSQL Dashboard

C’est tout pour aujourd’hui ! Je ferai un autre article prochainement pour présenter les fonctionnalités de Temboard notamment le tableau de bord, la versionnement des fichiers postgresql.conf, comment tuer une transaction, etc.

En attendant si vous prenez le temps de tester cet outil, envoyez nous un message à contact@dalibo.com et dites-nous ce que vous en pensez, quelle fonctionnalités vous manquent !

Temboard est encore un projet très jeune et nous avons prévu d’ajouter plus de plugins dans les mois à venir !

par Damien Clochard le dimanche 5 février 2017 à 10h17

jeudi 2 février 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 29 janvier 2017

Vous pouvez bénéficier de 50% de remise pour le SCALE 15x avec le code PGDAY : https://www.socallinuxexpo.org/scale/15x

[ndt: Meetup à Lyon le 21 février :https://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170129225258.GC17445@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Tom Lane pushed:

  • Volatile-ize some plperl variables that must survive into PG_CATCH blocks. This appears to be necessary to fix a failure seen on buildfarm member sittella. It shouldn't be necessary according to the letter of the C standard, because we don't change the values of these variables within the PG_TRY blocks; but somehow gcc 4.7.2 is dropping the ball. Discussion: https://postgr.es/m/17555.1485179975@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c0ef456b9714215cae0ce3561c7c83629a7301cf
  • Fix example plan in optimizer/README. Joining three tables only takes two join nodes. I think when I (tgl) wrote this, I was envisioning possible additional joins; but since the example doesn't show any fourth table, it's just confusing to write a third join node. Etsuro Fujita Discussion: https://postgr.es/m/e6cfbaa3-af02-1abc-c25e-8fa5c6bc4e21@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/3c821466abcdb8120ab0dfbce02f3bbae3b62025
  • Allow password file name to be specified as a libpq connection parameter. Formerly an alternate password file could only be selected via the environment variable PGPASSFILE; now it can also be selected via a new connection parameter "passfile", corresponding to the conventions for most other connection parameters. There was some concern about this creating a security weakness, but it was agreed that that argument was pretty thin, and there are clear use-cases for handling password files this way. Julian Markwort, reviewed by Fabien Coelho, some adjustments by me Discussion: https://postgr.es/m/a4b4f4f1-7b58-a0e8-5268-5f7db8e8ccaa@uni-muenster.de http://git.postgresql.org/pg/commitdiff/ba005f193d88a8404e81db3df223cf689d64d75e
  • Use non-conflicting table names in new regression test case. Commit 587cda35c added a test to updatable_views.sql that created tables named the same as tables used by the concurrent inherit.sql script. Unsurprisingly, this results in random failures. Pick different names. Per buildfarm. http://git.postgresql.org/pg/commitdiff/7fa7bf18e493e130147e62cf7dc33010f164126c
  • Improve speed of contrib/postgres_fdw regression tests. Commit 7012b132d added some tests that consumed an excessive amount of time, more than tripling the time needed for "make installcheck" for this module. Add filter conditions to reduce the number of rows scanned, bringing the runtime down to within hailing distance of what it was before. Jeevan Chalke and Ashutosh Bapat, per a gripe from me Discussion: https://postgr.es/m/16565.1478104765@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/aa7f593b1ffa9717bd5570174944c06c482d1c1f
  • Change unknown-type literals to type text in SELECT and RETURNING lists. Previously, we left such literals alone if the query or subquery had no properties forcing a type decision to be made (such as an ORDER BY or DISTINCT clause using that output column). This meant that "unknown" could be an exposed output column type, which has never been a great idea because it could result in strange failures later on. For example, an outer query that tried to do any operations on an unknown-type subquery output would generally fail with some weird error like "failed to find conversion function from unknown to text" or "could not determine which collation to use for string comparison". Also, if the case occurred in a CREATE VIEW's query then the view would have an unknown-type column, causing similar failures in queries trying to use the view. To fix, at the tail end of parse analysis of a query, forcibly convert any remaining "unknown" literals in its SELECT or RETURNING list to type text. However, provide a switch to suppress that, and use it in the cases of SELECT inside a set operation or INSERT command. In those cases we already had type resolution rules that make use of context information from outside the subquery proper, and we don't want to change that behavior. Also, change creation of an unknown-type column in a relation from a warning to a hard error. The error should be unreachable now in CREATE VIEW or CREATE MATVIEW, but it's still possible to explicitly say "unknown" in CREATE TABLE or CREATE (composite) TYPE. We want to forbid that because it's nothing but a foot-gun. This change creates a pg_upgrade failure case: a matview that contains an unknown-type column can't be pg_upgraded, because reparsing the matview's defining query will now decide that the column is of type text, which doesn't match the cstring-like storage that the old materialized column would actually have. Add a checking pass to detect that. While at it, we can detect tables or composite types that would fail, essentially for free. Those would fail safely anyway later on, but we might as well fail earlier. This patch is by me, but it owes something to previous investigations by Rahila Syed. Also thanks to Ashutosh Bapat and Michael Paquier for review. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/1e7c4bb0049732ece651d993d03bb6772e5d281a
  • Make UNKNOWN into an actual pseudo-type. Previously, type "unknown" was labeled as a base type in pg_type, which perhaps had some sense to it because you were allowed to create tables with unknown-type columns. But now that we don't allow that, it makes more sense to label it a pseudo-type. This has the additional effects of forbidding use of "unknown" as a domain base type, cast source or target type, PL function argument or result type, or plpgsql local variable type; all of which seem like good holes to plug. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/d8d32d9a56a3cecfb14e8f47ebd50b780edffe60
  • Remove vestigial resolveUnknown arguments from transformSortClause etc. There's really no situation where we don't want these unknown-to-text conversions to happen. The alternative is failure anyway, and the one caller that was passing "false" did so only because it expected the case could not arise. Might as well simplify the code. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/aebeb4790c750dc808c1c5afb3cb435116244e36
  • Introduce convenience macros to hide JsonbContainer header accesses better. This improves readability a bit and may make future improvements easier. In passing, make sure that the JB_ROOT_IS_XXX macros deliver boolean (0/1) results; the previous coding was a bug hazard, though no actual bugs are known. Nikita Glukhov, extended a bit by me Discussion: https://postgr.es/m/9e21a39c-c1d7-b9b5-44a0-c5345a5029f6@postgrespro.ru http://git.postgresql.org/pg/commitdiff/f7c62462402972b13d10e43f104ca0c0fecb6d08
  • Ensure that a tsquery like '!foo' matches empty tsvectors. !foo means "the tsvector does not contain foo", and therefore it should match an empty tsvector. ts_match_vq() overenthusiastically supposed that an empty tsvector could never match any query, so it forcibly returned FALSE, the wrong answer. Remove the premature optimization. Our behavior on this point was inconsistent, because while seqscans and GIST index searches both failed to match empty tsvectors, GIN index searches would find them, since GIN scans don't rely on ts_match_vq(). That makes this certainly a bug, not a debatable definition disagreement, so back-patch to all supported branches. Report and diagnosis by Tom Dunstan (bug #14515); added test cases by me. Discussion: https://postgr.es/m/20170126025524.1434.97828@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/9d4ca01314ba0c571139c5560a40ef764ad0feee
  • Use castNode() in a bunch of statement-list-related code. When I wrote commit ab1f0c822, I really missed the castNode() macro that Peter E. had proposed shortly before. This back-fills the uses I would have put it to. It's probably not all that significant, but there are more assertions here than there were before, and conceivably they will help catch any bugs associated with those representation changes. I left behind a number of usages like "(Query *) copyObject(query_var)". Those could have been converted as well, but Peter has proposed another notational improvement that would handle copyObject cases automatically, so I let that be for now. http://git.postgresql.org/pg/commitdiff/7afd56c3c6d8360a5bfdfb2de30038b239fd756b
  • Orthography fixes for new castNode() macro. Clean up hastily-composed comment. Normalize whitespace. Erik Rijkers and myself http://git.postgresql.org/pg/commitdiff/fefb86b14776321ac153836398eadde867ff31af
  • Improve comments about ProcessUtility's queryString parameter. Per discussion with Craig Ringer. http://git.postgresql.org/pg/commitdiff/fde5c037925b01b937923606c39460d94965672e
  • Restructure hba.c to replace 3 parallel lists with single list of structs. tokenize_file() now returns a single list of TokenizedLine structs, carrying the same information as before. We were otherwise going to grow a fourth list to deal with error messages, and that was getting a bit silly. Haribabu Kommi, revised a bit by me Discussion: https://postgr.es/m/CAJrrPGfbgbKsjYp=bgZXhMcgxoaGSoBb9fyjrDoOW_YymXv1Kw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/350cb921ae2ced8674e098d0645f2493e5757ad1

Ãlvaro Herrera pushed:

Tatsuo Ishii pushed:

Fujii Masao pushed:

Robert Haas pushed:

Simon Riggs pushed:

Andres Freund pushed:

Correctifs en attente

Michaël Paquier sent in a patch to enable replication connections by default in pg_hba.conf.

Michaël Paquier sent in another revision of a patch to ensure that pg_dump and pg_dumpall sync their output to disk by default.

Michaël Paquier sent in a patch to ensure that launching replication workers is a NOOP when wal_level < logical, and to set the default number of such workers to 0.

Craig Ringer sent in three more revisions of a patch to fix a race between clog truncation and lookup, and introduce txid_status(bigint) to get status of an xact.

Petr Jelínek sent in two more revisions of a patch to use the asynchronous connect API in libpqwalreceiver, close the replication connection when slot creation gets canceled, ensure that stringinfo buffers in walsender are always initialized, fix after trigger execution in logical replication, and add RENAME support for PUBLICATIONs and SUBSCRIPTIONs.

Amit Kapila sent in another revision of a patch to implement parallel index scans.

Dilip Kumar sent in three more revisions of a patch to implement parallel bitmap heap scans.

Beena Emerson sent in another revision of a patch to enable increasing the default WAL segment size.

Corey Huinker sent in three more revisions of a patch to add \if and friends to psql.

Etsuro Fujita sent in another revision of a patch to fix a bug in the PostgreSQL FDW.

Ivan Kartyshov sent in another revision of a patch to make async slave to wait for lsn to be replayed.

Dagfinn Ilmari Mannsåker sent in another revision of a patch to add GUCs for predicate lock promotion thresholds.

Craig Ringer sent in another revision of a patch to implement logical decoding on standby.

Nico Williams sent in a patch to implement an expanded version of materialized views and a contrib extension.

Nico Williams sent in a patch to implement pqasyncnotifier.c, a shell command client for LISTEN.

Amit Langote sent in a patch to add relkind checks to certain contrib modules.

Kyotaro HORIGUCHI sent in another revision of a patch to clean up the negative cache of pg_statistic when dropping a relation and of pg_class when dropping a schema.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Ashutosh Sharma sent in two more revisions of a patch to add pgstathashindex() to pgstattuple extension.

Jim Nasby sent in another revision of a patch to add faster methods for getting SPI results.

Claudio Freire sent in two more revisions of a patch to allow usage of more than 1GB of work mem in VACUUM.

Pavel Stěhule sent in two revisions of a patch to enable forcing a custom or generic plan in PL/pgsql.

Stas Kelvich and Nikhil Sontakke traded patches to speed up two-phase transactions.

Haribabu Kommi sent in three more revisions of a patch to implement a 64-bit (EUI-64) macaddr data type.

Haribabu Kommi and Tom Lane traded patches to implement a pg_hba_file_settings view.

Peter Eisentraut sent in another revision of a patch to add ICU support.

Beena Emerson sent in a patch to add tab completion to ALTER in psql.

Daniel Vérité sent in another revision of a patch to improve psql hooks for variables.

Nikita Glukhov sent in another revision of a patch to make a recursive version of json_populate_record().

Ãlvaro Herrera sent in another revision of a patch to implement xmltable().

Fabien COELHO sent in two more revisions of a patch to add more functions and operators to pgbench.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Michaël Paquier sent in three revisions of a patch to refactor the replication commands output.

Vladimir Rusinov sent in a patch to rename the sql-callable functions with xlog to have wal instead.

Etsuro Fujita sent in another revision of a patch to push down more UPDATEs/DELETEs in postgres_fdw.

Ashutosh Bapat sent in another revision of a patch to speed up aggregate pushdown tests.

Julian Markwort sent in a patch to extended the functionality of pg_stat_statements so it can track worst and best case execution plans.

Fabien COELHO sent in a patch to fix an infelicity between pgbench's --connect and --rate options.

Kyotaro HORIGUCHI sent in another revision of a patch to use a radix tree for character conversion.

Peter Eisentraut sent in two more revisions of a patch to add test coverage for sequences.

David Fetter and Corey Huinker traded patches to add copy_srf(), a set-returning function corresponding to COPY IN.

Masahiko Sawada sent in two more revisions of a patch to support 2PC in FDWs.

Michaël Paquier sent in two revisions of a patch to remove race conditions between the checkpointer and the init fork creations by making index init forks go through the shared buffers instead of having their empty() routines handle the flush of the page created.

Robert Haas sent in a patch to remove some hard-coded superuser checks.

Robert Haas sent in another revision of a patch to rename things *xlog* to the corresponding *wal*.

Mithun Cy sent in two more revisions of a patch to cache hash index meta pages.

David Rowley sent in three more revisions of a patch to improve performance where for joins where the outer side is unique.

Venkata B Nagothi sent in a patch to generate an error by aborting the recovery process instead of starting up the cluster if the intended recovery target point is not reached, and give an option to DBA to resume the recovery process from exactly where it stopped.

Ashutosh Sharma sent in another revision of a patch to add microvacuum support for hash indexes.

Noah Misch sent in another revision of a patch to remove link-time cross-module refs in contrib.

Christoph Berg sent in a patch to use \G to use expanded output for a query or current query buffer.

Robert Haas and Ãlvaro Herrera traded patches to add hash index support to to the pageinspect contrib extension.

Thomas Munro sent in another revision of a patch to implement parallel shared hash.

David Rowley sent in another revision of a patch to fix an infelicity between CONNECTION LIMIT and Parallel Query.

Tom Lane sent in a patch to create a separate test file for exercising system views.

par chl le jeudi 2 février 2017 à 01h53

Nouvelles hebdomadaires de PostgreSQL - 22 janvier 2017

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170122232424.GA6120@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Fujii Masao pushed:

  • Fix typos in comments. Masahiko Sawada http://git.postgresql.org/pg/commitdiff/8fa6019b405f9d39539a77c6f5f11fe640ddf955
  • Fix an assertion failure related to an exclusive backup. Previously multiple sessions could execute pg_start_backup() and pg_stop_backup() to start and stop an exclusive backup at the same time. This could trigger the assertion failure of "FailedAssertion("!(XLogCtl->Insert.exclusiveBackup)". This happend because, even while pg_start_backup() was starting an exclusive backup, other session could run pg_stop_backup() concurrently and mark the backup as not-in-progress unconditionally. This patch introduces ExclusiveBackupState indicating the state of an exclusive backup. This state is used to ensure that there is only one session running pg_start_backup() or pg_stop_backup() at the same time, to avoid the assertion failure. Back-patch to all supported versions. Author: Michael Paquier Reviewed-By: Kyotaro Horiguchi and me Reported-By: Andreas Seltenreich Discussion: <87mvktojme.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/974ece58bbb3c0ef185a9d44b1cedae51cd56b04
  • Add description of temporary column into pg_replication_slots doc. Ayumi Ishii http://git.postgresql.org/pg/commitdiff/954737095061e5b5f1d87fb8cc43f7f8afff64c6

Magnus Hagander pushed:

Tom Lane pushed:

  • Fix NULL pointer dereference in tuplesort.c. Oversight in commit e94568ecc. This could cause a crash when an external datum tuplesort of a pass-by-value type required multiple passes. Per report from Mithun Cy. Peter Geoghegan Discussion: https://postgr.es/m/CAD__OujuhfWFULGFSt1fyHqUb8N-XafjJhudwt88V0Qs2o84qg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4e46c97fde42fa8ca57d29b9b47f2ebd11ab8105
  • Fix check_srf_call_placement() to handle VALUES cases correctly. INSERT ... VALUES with a single VALUES row is implemented quite differently from the general VALUES case. A user-visible implication of that is that we accept SRFs in the single-row case, but not in the multi-row case. That's a historical artifact no doubt, but in view of the lack of field complaints, I'm not excited about fixing it right now. However, check_srf_call_placement() needs to know about this, first because it should throw an error in the unsupported case, and second because it should set p_hasTargetSRFs in the single-row case (because we treat that like a SELECT tlist). That's an oversight in commit a4c35ea1c. To fix, split EXPR_KIND_VALUES into two values. So far as I can see, this is the only place where we need to distinguish the two cases at present; but there might be more later. Patch by me, per report from Andres Freund. Discussion: https://postgr.es/m/20170116081548.zg63zltblwimpfgp@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/d43a619c60405ecda275ca9e3ac9ead242e20ecb
  • Improve RLS planning by marking individual quals with security levels. In an RLS query, we must ensure that security filter quals are evaluated before ordinary query quals, in case the latter contain "leaky" functions that could expose the contents of sensitive rows. The original implementation of RLS planning ensured this by pushing the scan of a secured table into a sub-query that it marked as a security-barrier view. Unfortunately this results in very inefficient plans in many cases, because the sub-query cannot be flattened and gets planned independently of the rest of the query. To fix, drop the use of sub-queries to enforce RLS qual order, and instead mark each qual (RestrictInfo) with a security_level field establishing its priority for evaluation. Quals must be evaluated in security_level order, except that "leakproof" quals can be allowed to go ahead of quals of lower security_level, if it's helpful to do so. This has to be enforced within the ordering of any one list of quals to be evaluated at a table scan node, and we also have to ensure that quals are not chosen for early evaluation (i.e., use as an index qual or TID scan qual) if they're not allowed to go ahead of other quals at the scan node. This is sufficient to fix the problem for RLS quals, since we only support RLS policies on simple tables and thus RLS quals will always exist at the table scan level only. Eventually these qual ordering rules should be enforced for join quals as well, which would permit improving planning for explicit security-barrier views; but that's a task for another patch. Note that FDWs would need to be aware of these rules --- and not, for example, send an insecure qual for remote execution --- but since we do not yet allow RLS policies on foreign tables, the case doesn't arise. This will need to be addressed before we can allow such policies. Patch by me, reviewed by Stephen Frost and Dean Rasheed. Discussion: https://postgr.es/m/8185.1477432701@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/215b43cdc8d6b4a1700886a39df1ee735cb0274d
  • Avoid conflicts with collation aliases generated by stripping. This resulted in failures depending on the order of "locale -a" output. The original coding in initdb sorted the results, but that should be unnecessary as long as "locale -a" doesn't print duplicate names. The original entries will then all be non-dups, and while we might generate duplicate aliases by stripping, they should be for different encodings and thus not conflict. Even if the latter assumption fails somehow, it won't be fatal because we're using if_not_exists mode for the aliases. Discussion: https://postgr.es/m/26116.1484751196%40sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0333a7340054c3356940759b1ab2391eed572171
  • Disable transforms that replaced AT TIME ZONE with RelabelType. These resulted in wrong answers if the relabeled argument could be matched to an index column, as shown in bug #14504 from Evgeniy Kozlov. We might be able to resurrect these optimizations by adjusting the planner's treatment of RelabelType, or by adjusting btree's rules for selecting comparison functions, but either solution will take careful analysis and does not sound like a fit candidate for backpatching. I left the catalog infrastructure in place and just reduced the transform functions to always-return-NULL. This would be necessary anyway in the back branches, and it doesn't seem important to be more invasive in HEAD. Bug introduced by commit b8a18ad48. Back-patch to 9.5 where that came in. Report: https://postgr.es/m/20170118144828.1432.52823@wrigleys.postgresql.org Discussion: https://postgr.es/m/18771.1484759439@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c22ecc6562aac895f0f0529707d7bdb460fd2a49
  • Reset the proper GUC in create_index test. Thinko in commit a4523c5aa. It doesn't really affect anything at present, but it would be a problem if any tests added later in this file ought to get index-only-scan plans. Back-patch, like the previous commit, just to avoid surprises in case we add such a test and then back-patch it. Nikita Glukhov Discussion: https://postgr.es/m/8b70135d-ad38-bdd8-ac92-71e2b3c273cf@postgrespro.ru http://git.postgresql.org/pg/commitdiff/1586317c3f57e619e0cde674c6da406f9d73aaff
  • Doc: improve documentation of new SRF-in-tlist behavior. Correct a misstatement about how things used to work: we did allow nested SRFs before, as long as no function had more than one set-returning input. Also, attempt to document the fact that the new implementation changes the behavior for SRFs within conditional constructs (eg CASE): the conditional construct no longer gates whether the SRF is run, and thus cannot affect the number of rows emitted. We might want to change this behavior, but first it behooves us to see if we can explain it. Minor other wordsmithing on what I wrote yesterday, too. Discussion: https://postgr.es/m/20170118214702.54b2mdbxce5piwv5@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/f13a1277aa2df301583c6db9c2989d2e9d7f6483
  • Fix Assert failure induced by commit 215b43cdc. I'd somehow talked myself into believing that set_append_rel_size doesn't need to worry about getting back an AND clause when it applies eval_const_expressions to the result of adjust_appendrel_attrs (that is, transposing the appendrel parent's restriction clauses for one child). But that is nonsense, and Andreas Seltenreich's fuzz tester soon turned up a counterexample. Put back the make_ands_implicit step that was there before, and add a regression test covering the case. Report: https://postgr.es/m/878tq6vja6.fsf@ansel.ydns.eu http://git.postgresql.org/pg/commitdiff/d479e37e3d20efad8b178e0f1e468c086a7519a8
  • Avoid core dump for empty prepared statement in an aborted transaction. Brown-paper-bag bug in commit ab1f0c822: the old code here coped with null CachedPlanSource.raw_parse_tree, the new code not so much. Per report from Dave Cramer. No regression test, because our core testing infrastructure doesn't provide any easy way to exercise this path. Fortunately, the JDBC crew test it regularly. Discussion: https://postgr.es/m/CADK3HH+Ug3xCysKqw_dZOnaNnytZ1Rh5yP05hjO-e4NoyRxVvA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/ba61a04bc7fefeee03416d9911eb825c4897c223
  • Allow backslash line continuations in pgbench's meta commands. A pgbench meta command can now be continued onto additional line(s) of a script file by writing backslash-return. The continuation marker is equivalent to white space in that it separates tokens. Eventually it'd be nice to have the same thing in psql, but that will be a much larger project. Fabien Coelho, reviewed by Rafia Sabih Discussion: https://postgr.es/m/alpine.DEB.2.20.1610031049310.19411@lancre http://git.postgresql.org/pg/commitdiff/cdc2a70470bdbe3663dc464deb753d6d931bba61
  • Try to fix non-MSVC Windows builds in the wake of logical replication. pgoutput evidently needs to be built without -DBUILDING_DLL. (It seems like a pretty bad idea that these makefiles need to know exactly where all the shlibs are in the tree, or maybe what's bad is putting them under src/backend/. But right now is not the time to redesign that.) Also, remove "override CPPFLAGS" in pgoutput's Makefile. I don't think that that actually has any bad consequences, but it's certainly useless in a directory that has no .h files, and it might be contributing to the failure somehow. Per buildfarm. http://git.postgresql.org/pg/commitdiff/0502e854640bd024f349c0df46b7dd6812b8c05c
  • Fix cross-shlib linking in temporary installs on HPUX 10. Turns out this has been broken for years and we'd not noticed. The one case that was getting exercised in the buildfarm, or probably anywhere else, was postgres_fdw.sl's reference to libpq.sl; and it turns out that that was always going to libpq.sl in the actual installation directory not the temporary install. We'd not noticed because the buildfarm script does "make install" before it tests contrib. However, the recent addition of a logical-replication test to the core regression scripts resulted in trying to use libpqwalreceiver.sl before "make install" happens, and that failed for lack of finding libpq.sl, as shown by failures on buildfarm members gaur and pademelon. There are two changes needed to fix it: the magic environment variable to specify shlib search path at runtime is SHLIB_PATH not LD_LIBRARY_PATH, and the shlib link command needs to specify the +s switch else the library will not honor SHLIB_PATH. I'm not quite sure why buildfarm members anole and gharial (HPUX 11) didn't show the same failure. Consulting man pages on the web says that HPUX 11 honors both LD_LIBRARY_PATH and SHLIB_PATH, which would explain half of it, and the rather confusing wording I've been able to find suggests that +s might effectively be the default in HPUX 11. But it seems at least as likely that there's just a libpq.so installed in /usr/lib on that machine; as long as it's not too ancient, that would satisfy the test. In any case I do not think this patch will break HPUX 11. At the moment I don't see a need to back-patch this, since it only matters for testing purposes, not to mention that HPUX 10 is probably dead in the real world anyway. http://git.postgresql.org/pg/commitdiff/d2ab1176160e30543da1e48f7e0d17564852b693
  • Remove no-longer-needed loop in ExecGather(). Coverity complained quite properly that commit ea15e1867 had introduced unreachable code into ExecGather(); to wit, it was no longer possible to iterate the final for-loop more or less than once. So remove the for(). In passing, clean up a couple of comments, and make better use of a local variable. http://git.postgresql.org/pg/commitdiff/0a8b9d3b2c57028f7100078cd711370f396d5a81
  • Relocate static function declarations to be after typedefs in jsonfuncs.c. Project style is to put things in this order, for the good and sufficient reason that you often need the typedefs in the function declarations. There already was one function declaration that needed a typedef, which was randomly placed away from all the other static function declarations in consequence. And the submitted patch for better json_populate_record functionality jumped through even more hoops in order to preserve this bad idea. This patch only moves lines from point A to point B, no other changes. http://git.postgresql.org/pg/commitdiff/90992e0e2f9fc4aa0f6402f0327604e5fef4630c

Peter Eisentraut pushed:

Álvaro Herrera pushed:

Robert Haas pushed:

Andres Freund pushed:

  • Move targetlist SRF handling from expression evaluation to new executor node. Evaluation of set returning functions (SRFs_ in the targetlist (like SELECT generate_series(1,5)) so far was done in the expression evaluation (i.e. ExecEvalExpr()) and projection (i.e. ExecProject/ExecTargetList) code. This meant that most executor nodes performing projection, and most expression evaluation functions, had to deal with the possibility that an evaluated expression could return a set of return values. That's bad because it leads to repeated code in a lot of places. It also, and that's my (Andres's) motivation, made it a lot harder to implement a more efficient way of doing expression evaluation. To fix this, introduce a new executor node (ProjectSet) that can evaluate targetlists containing one or more SRFs. To avoid the complexity of the old way of handling nested expressions returning sets (e.g. having to pass up ExprDoneCond, and dealing with arguments to functions returning sets etc.), those SRFs can only be at the top level of the node's targetlist. The planner makes sure (via split_pathtarget_at_srfs()) that SRF evaluation is only necessary in ProjectSet nodes and that SRFs are only present at the top level of the node's targetlist. If there are nested SRFs the planner creates multiple stacked ProjectSet nodes. The ProjectSet nodes always get input from an underlying node. We also discussed and prototyped evaluating targetlist SRFs using ROWS FROM(), but that turned out to be more complicated than we'd hoped. While moving SRF evaluation to ProjectSet would allow to retain the old "least common multiple" behavior when multiple SRFs are present in one targetlist (i.e. continue returning rows until all SRFs are at the end of their input at the same time), we decided to instead only return rows till all SRFs are exhausted, returning NULL for already exhausted ones. We deemed the previous behavior to be too confusing, unexpected and actually not particularly useful. As a side effect, the previously prohibited case of multiple set returning arguments to a function, is now allowed. Not because it's particularly desirable, but because it ends up working and there seems to be no argument for adding code to prohibit it. Currently the behavior for COALESCE and CASE containing SRFs has changed, returning multiple rows from the expression, even when the SRF containing "arm" of the expression is not evaluated. That's because the SRFs are evaluated in a separate ProjectSet node. As that's quite confusing, we're likely to instead prohibit SRFs in those places. But that's still being discussed, and the code would reside in places not touched here, so that's a task for later. There's a lot of, now superfluous, code dealing with set return expressions around. But as the changes to get rid of those are verbose largely boring, it seems better for readability to keep the cleanup as a separate commit. Author: Tom Lane and Andres Freund Discussion: https://postgr.es/m/20160822214023.aaxz5l4igypowyri@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/69f4b9c85f168ae006929eec44fc44d569e846b9
  • Adapt python regression tests to 69f4b9c85f16. Hopefully this'll unbreak the buildfarm. http://git.postgresql.org/pg/commitdiff/8b07aee8c5d803801c00103f0d61e32356aaf29c
  • Remove obsoleted code relating to targetlist SRF evaluation. Since 69f4b9c plain expression evaluation (and thus normal projection) can't return sets of tuples anymore. Thus remove code dealing with that possibility. This will require adjustments in external code using ExecEvalExpr()/ExecProject() - that should neither be hard nor very common. Author: Andres Freund and Tom Lane Discussion: https://postgr.es/m/20160822214023.aaxz5l4igypowyri@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/ea15e18677fc2eff3135023e27f69ed8821554ed
  • Fix platform dependant regression output triggered by 69f4b9c85f16. Due to the changed costing in that commit hash-aggregates started to be used, which results in big-endian vs. little-endian output differences. Disable hash-aggs for those tests. Author: Andres Freund, with input from Tom Lane Discussion: https://postgr.es/m/22891.1484791792@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/182200531a92204b0547d337f50b665d222af168

Stephen Frost pushed:

  • Dump sequence data based on the TableDataInfo flag. When considering a sequence's Data entry in dumpSequenceData, we were actually looking at the sequence definition's dump flag to decide if we should dump the data or not. That's generally fine, except for when the sequence data entry was created by processExtensionTables() because it's a config sequence. In that case, the sequence itself won't be marked as dumping data because it's part of an extension, leading to the need for processExtensionTables() to create the sequence data entry. This leads to extension config sequence data not being included in the dump when it should be. Fix this by looking at the sequence data's dump flag instead, just as dumpTableData() was doing for tables (which is why config tables were correctly being handled), and add a regression test to make sure we don't break it moving forward. All of this is a bit round-about since we can now represent which components of a given dump item should be dumped out through the dump flag. A future improvement might be to change checkExtensionMembership() to check for config sequences/tables and set the dump flag based on that directly, possibly removing the need for processExtensionTables(). Bug found by Daniele Varrazzo. Discussion: https://postgr.es/m/CA+mi_8ZmxQM7+nZ7pJ8uyfxc9V3o=UAG14dVqvftdmvw8OJ3gQ@mail.gmail.com Patch by Michael Paquier, with some tweaking of the regression tests by me. Back-patch to 9.6 where the bug was introduced. http://git.postgresql.org/pg/commitdiff/bec96c82f8ff4fcf7ef0f070f6f7447edf106de3

Correctifs en attente

Michaël Paquier and Karl O. Pinc traded patches to implement pg_current_logfile().

Michaël Paquier sent in another revision of a patch to rename pg_clog to pg_xact and pg_subtrans to pg_subxact.

Masahiko Sawada sent in two more revisions of a patch to support FDW transactions.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Haribabu Kommi sent in four more revisions of a patch to implement a pg_hba_file_settings view.

Beena Emerson and Robert Haas traded patches to enable increasing WAL segment size.

Amit Kapila and Robert Haas traded patches to enable parallel index scans.

Michaël Paquier sent in a flock of back-patches for an issue in backups on standbys uncovered by sqlsmith.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Nikita Glukhov sent in a patch to implement K-nearest-neighbor for B-Tree indexes.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Magnus Hagander and Michaël Paquier traded patches to implement a version of jsonb_delete which takes arrays as input.

Haribabu Kommi sent in another revision of a patch to implement a pg_stat_sql view.

Etsuro Fujita sent in another revision of a patch to fix a bug in foreign joins.

David Rowley sent in two more revisions of a patch to improve performance for outer joins where the outer side is unique.

Haribabu Kommi sent in another revision of a patch to implement a 64-bit macaddr type.

Ashutosh Sharma sent in another revision of a patch to add pgstathashindex to the pgstattuple extension.

Ashutosh Sharma and Jesper Pedersen traded patches to add hash index support to pageinspect.

Petr Jelínek sent in two more revisions of a patch to enable logical replication support for the initial data copy.

Petr Jelínek sent in another revision of a patch to add RENAME support for PUBLICATIONs and SUBSCRIPTIONs.

Petr Jelínek sent in a flock of patches to fix issues in logical replication.

Rafia Sabih sent in another revision of a patch to implement parallel index-only scans.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Dmitry Fedin sent in a patch to change a misleading comment in pgtime.h.

Yugo Nagata sent in a patch to fix a comment in freelist.c.

Erik Rijkers sent in a patch to improve the logical replication documentation.

Jim Nasby sent in a patch to protect syscache from bloating with negative cache entries.

Petr Jelínek sent in a patch to fix AFTER trigger execution in logical replication.

Tom Lane sent in another revision of a patch to prevent the possibility of UNKNOWN output columns.

par chl le jeudi 2 février 2017 à 01h43

jeudi 19 janvier 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 15 janvier 2017

[ndt: Meetup à Toulouse le 2 mars :https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Toulouse/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170116044539.GA14726@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Magnus Hagander pushed:

Tom Lane pushed:

  • Expand the regression tests for PL/Tcl. This raises the test coverage (by line count) in pltcl.c from about 70% to 86%. Karl Lehenbauer and Jim Nasby Discussion: https://postgr.es/m/92a1670d-21b6-8f03-9c13-e4fb2207ab7b@BlueTreble.com http://git.postgresql.org/pg/commitdiff/961bed0208912a929a47c5a30190ff76748f3a03
  • Fix error handling in pltcl_returnnext. We can't throw elog(ERROR) out of a Tcl command procedure; we have to catch the error and return TCL_ERROR to the Tcl interpreter. pltcl_returnnext failed to meet this requirement, so that errors detected by pltcl_build_tuple_result or other functions called here led to longjmp'ing out of the Tcl interpreter and thereby leaving it in a bad state. Use the existing subtransaction support to prevent that. Oversight in commit 26abb50c4, found more or less accidentally by the buildfarm thanks to the tests added in 961bed020. Report: https://postgr.es/m/30647.1483989734@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/8c5722948e831c1862a39da2bb5d793a6f2aabab
  • Fix field order in struct catcache. Somebody failed to grasp the point of having the #ifdef CATCACHE_STATS fields at the end of the struct. Put that back the way it should be, and add a comment making it more explicit why it should be that way. http://git.postgresql.org/pg/commitdiff/e2117e4ab0d7fcb76f20cbf2e52541998d639d9e
  • In PL/Tcl tests, don't choke if optional error fields are missing. This fixes a portability issue introduced by commit 961bed020: with a compiler that doesn't support PG_FUNCNAME_MACRO, the "funcname" field of errorCode won't be provided, leading to a failure of the unset command. I added -nocomplain to the unset commands for filename and lineno too, just in case, though I know of no platform that wouldn't populate those fields. (BTW, -nocomplain is new in Tcl 8.4, but fortunately we dropped support for pre-8.4 Tcl some time ago.) Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/5b29e6b688d1e783a098aa24f9e795f8de798a87
  • Fix some more regression test row-order-instability issues. Commit 0563a3a8b just introduced another instance of the same unsafe testing methodology that appeared in 2ac3ef7a0, which I corrected in 257d81572. Robert/Amit, please stop doing that. Also look through the rest of f0e44751d's test cases, and correct some other queries with underdetermined ordering of results from the system catalogs. These haven't failed in the buildfarm yet, but I don't have any confidence in that staying true. Per multiple buildfarm members. http://git.postgresql.org/pg/commitdiff/5ad966ab1c50e829462f2b3e3ffa59e2d95479e6
  • Throw suitable error for COPY TO STDOUT/FROM STDIN in a SQL function. A client copy can't work inside a function because the FE/BE wire protocol doesn't support nesting of a COPY operation within query results. (Maybe it could, but the protocol spec doesn't suggest that clients should support this, and libpq for one certainly doesn't.) In most PLs, this prohibition is enforced by spi.c, but SQL functions don't use SPI. A comparison of _SPI_execute_plan() and init_execution_state() shows that rejecting client COPY is the only discrepancy in what they allow, so there's no other similar bugs. This is an astonishingly ancient oversight, so back-patch to all supported branches. Report: https://postgr.es/m/BY2PR05MB2309EABA3DEFA0143F50F0D593780@BY2PR05MB2309.namprd05.prod.outlook.com http://git.postgresql.org/pg/commitdiff/75abb955dfef064f2fbc5c043f37fff8d0262ffe
  • Change representation of statement lists, and add statement location info. This patch makes several changes that improve the consistency of representation of lists of statements. It's always been the case that the output of parse analysis is a list of Query nodes, whatever the types of the individual statements in the list. This patch brings similar consistency to the outputs of raw parsing and planning steps: * The output of raw parsing is now always a list of RawStmt nodes; the statement-type-dependent nodes are one level down from that. * The output of pg_plan_queries() is now always a list of PlannedStmt nodes, even for utility statements. In the case of a utility statement, "planning" just consists of wrapping a CMD_UTILITY PlannedStmt around the utility node. This list representation is now used in Portal and CachedPlan plan lists, replacing the former convention of intermixing PlannedStmts with bare utility-statement nodes. Now, every list of statements has a consistent head-node type depending on how far along it is in processing. This allows changing many places that formerly used generic "Node *" pointers to use a more specific pointer type, thus reducing the number of IsA() tests and casts needed, as well as improving code clarity. Also, the post-parse-analysis representation of DECLARE CURSOR is changed so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained SELECT remains a child of the DeclareCursorStmt rather than getting flipped around to be the other way. It's now true for both Query and PlannedStmt that utilityStmt is non-null if and only if commandType is CMD_UTILITY. That allows simplifying a lot of places that were testing both fields. (I think some of those were just defensive programming, but in many places, it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.) Because PlannedStmt carries a canSetTag field, we're also able to get rid of some ad-hoc rules about how to reconstruct canSetTag for a bare utility statement; specifically, the assumption that a utility is canSetTag if and only if it's the only one in its list. While I see no near-term need for relaxing that restriction, it's nice to get rid of the ad-hocery. The API of ProcessUtility() is changed so that what it's passed is the wrapper PlannedStmt not just the bare utility statement. This will affect all users of ProcessUtility_hook, but the changes are pretty trivial; see the affected contrib modules for examples of the minimum change needed. (Most compilers should give pointer-type-mismatch warnings for uncorrected code.) There's also a change in the API of ExplainOneQuery_hook, to pass through cursorOptions instead of expecting hook functions to know what to pick. This is needed because of the DECLARE CURSOR changes, but really should have been done in 9.6; it's unlikely that any extant hook functions know about using CURSOR_OPT_PARALLEL_OK. Finally, teach gram.y to save statement boundary locations in RawStmt nodes, and pass those through to Query and PlannedStmt nodes. This allows more intelligent handling of cases where a source query string contains multiple statements. This patch doesn't actually do anything with the information, but a follow-on patch will. (Passing this information through cleanly is the true motivation for these changes; while I think this is all good cleanup, it's unlikely we'd have bothered without this end goal.) catversion bump because addition of location fields to struct Query affects stored rules. This patch is by me, but it owes a good deal to Fabien Coelho who did a lot of preliminary work on the problem, and also reviewed the patch. Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre http://git.postgresql.org/pg/commitdiff/ab1f0c8225714aaa18d2f9ca4f80cd009f145421
  • Teach contrib/pg_stat_statements to handle multi-statement commands better. Make use of the statement boundary info added by commit ab1f0c822 to let pg_stat_statements behave more sanely when multiple SQL queries are jammed into one query string. It now records just the relevant part of the source string, not the whole thing, for each individual query. Even when no multi-statement strings are involved, users may notice small changes in the output: leading and trailing whitespace and semicolons will be stripped from statements, which did not happen before. Also, significantly expand pg_stat_statements' regression test script. Fabien Coelho, reviewed by Craig Ringer and Kyotaro Horiguchi, some mods by me Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre http://git.postgresql.org/pg/commitdiff/83f2061dd037477ec8479ee160367840e203a722
  • Fix matching of boolean index columns to sort ordering. Normally, if we have a WHERE clause like "indexcol = constant", the planner will figure out that that index column can be ignored when determining whether the index has a desired sort ordering. But this failed to work for boolean index columns, because a condition like "boolcol = true" is canonicalized to just "boolcol" which does not give rise to an EquivalenceClass. Add a check to allow the same type of deduction to be made in this case too. Per a complaint from Dima Pavlov. Arguably this is a bug, but given the limited impact and the small number of complaints so far, I won't risk destabilizing plans in stable branches by back-patching. Patch by me, reviewed by Michael Paquier Discussion: https://postgr.es/m/1788.1481605684@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0777f7a2e8e0a51f0f60cfe164d538bb459bf9f2

Ãlvaro Herrera pushed:

  • BRIN revmap pages are not standard pages, and therefore we ought not to tell XLogRegisterBuffer the opposite, when writing XLog for a brin update that moves the index tuple to a different page. Otherwise, xlog insertion would try to "compress the hole" when producing a full-page image for it; but since we don't update pd_lower/upper, the hole covers the whole page. On WAL replay, the revmap page becomes empty and so the entire portion of the index is useless and needs to be recomputed. This is low-probability: a BRIN update only moves an index tuple to a different page when the summary tuple is larger than the existing one, which doesn't happen with fixed-width datatypes. Also, the revmap page must be first after a checkpoint. Report and patch: Kuntal Ghosh Bug is alleged to have detected by a WAL-consistency-checking tool. Discussion: https://postgr.es/m/CAGz5QCJ=00UQjScSEFbV=0qO5ShTZB9WWz_Fm7+Wd83zPs9Geg@mail.gmail.com I posted a test case demonstrating the problem, but I'm refraining from adding it to the test suite; if the WAL consistency tool makes it in, that will be a better way to catch this from regressing. (We should definitely have someting that causes not-same-page updates, though.) http://git.postgresql.org/pg/commitdiff/7403561c0f6a8c62b79b6ddf0364ae6c01719068
  • Fix ALTER TABLE / SET TYPE for irregular inheritance. If inherited tables don't have exactly the same schema, the USING clause in an ALTER TABLE / SET DATA TYPE misbehaves when applied to the children tables since commit 9550e8348b79. Starting with that commit, the attribute numbers in the USING expression are fixed during parse analysis. This can lead to bogus errors being reported during execution, such as: ERROR: attribute 2 has wrong type DETAIL: Table has type smallint, but query expects integer. Since it wouldn't do to revert to the original coding, we now apply a transformation to map the attribute numbers to the correct ones for each child. Reported by Justin Pryzby Analysis by Tom Lane; patch by me. Discussion: https://postgr.es/m/20170102225618.GA10071@telsasoft.com http://git.postgresql.org/pg/commitdiff/3957b58b8885441c8d03bc1cfc00e47cf8cd7975
  • Fix overflow check in StringInfo; add missing casts. A few thinkos I introduced in fa2fa9955280. Also, amend a similarly broken comment. Report by Daniel Vérité. Authors: Daniel Vérité, Ãlvaro Herrera Discussion: https://postgr.es/m/1706e85e-60d2-494e-8a64-9af1e1b2186e@manitou-mail.org http://git.postgresql.org/pg/commitdiff/42f50cb8fa9848bbbc6776bcea03293a6b28b2d4

Stephen Frost pushed:

  • Fix invalid-parallel-jobs error message. Including the program name twice is not helpful: -> pg_dump -j -1 pg_dump: pg_dump: invalid number of parallel jobs Correct by removing the progname from the exit_horribly() call used when validating the number of parallel jobs. Noticed while testing various pg_dump error cases. Back-patch to 9.3 where parallel pg_dump was added. http://git.postgresql.org/pg/commitdiff/2ef6fe9cbae9fe7789a35cbc5fa1bbf78c163d42
  • pg_dump: Strict names with no matching schema. When using pg_dump --strict-names and a schema pattern which doesn't match any schemas (eg: --schema='nonexistant*'), we were incorrectly throwing an error claiming no tables were found when, really, there were no schemas found: -> pg_dump --strict-names --schema='nonexistant*' pg_dump: no matching tables were found for pattern "nonexistant*" Fix that by changing the error message to say 'schemas' instead, since that is what we are actually complaining about. Noticed while testing pg_dump error cases. Back-patch to 9.6 where --strict-names and this error message were introduced. http://git.postgresql.org/pg/commitdiff/abfd0095c1e1a2e3fad2696516b64871895334ec
  • pg_restore: Don't allow non-positive number of jobs. pg_restore will currently accept invalid values for the number of parallel jobs to run (eg: -1), unlike pg_dump which does check that the value provided is reasonable. Worse, '-1' is actually a valid, independent, parameter (as an alias for --single-transaction), leading to potentially completely unexpected results from a command line such as: -> pg_restore -j -1 Where a user would get neither parallel jobs nor a single-transaction. Add in validity checking of the parallel jobs option, as we already have in pg_dump, before we try to open up the archive. Also move the check that we haven't been asked to run more parallel jobs than possible on Windows to the same place, so we do all the option validity checking before opening the archive. Back-patch all the way, though for 9.2 we're adding the Windows-specific check against MAXIMUM_WAIT_OBJECTS as that check wasn't back-patched originally. Discussion: https://www.postgresql.org/message-id/20170110044815.GC18360%40tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/e72059f3757594c5530ce321acdbe67f0da5da13

Robert Haas pushed:

  • Improve coding in _hash_addovflpage. Instead of relying on the page contents to know whether we have advanced from the primary bucket page to an overflow page, track that explicitly. Amit Kapila, per a complaint by me. http://git.postgresql.org/pg/commitdiff/e898437460f55b49623d1aea435cd92e0011d54d
  • Fix incorrect function name in comment. Amit Langote http://git.postgresql.org/pg/commitdiff/76568d37865c5c21ae154008b2c681e3e32ac880
  • Fix cardinality estimates for parallel joins. For a partial path, the cardinality estimate needs to reflect the number of rows we think each worker will see, rather than the total number of rows; otherwise, costing will go wrong. The previous coding got this completely wrong for parallel joins. Unfortunately, this change may destabilize plans for users of 9.6 who have enabled parallel query, but since 9.6 is still fairly new I'm hoping expectations won't be too settled yet. Also, this is really a brown-paper-bag bug, so leaving it unfixed for the entire lifetime of 9.6 seems unwise. Related reports (whose import I initially failed to recognize) by Tomas Vondra and Tom Lane. Discussion: http://postgr.es/m/CA+TgmoaDxZ5z5Kw_oCQoymNxNoVaTCXzPaODcOuao=CzK8dMZw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/0c2070cefa0e5d097b715c9a3b9b5499470019aa
  • Fix a bug in how we generate partition constraints. Move the code for doing parent attnos to child attnos mapping for Vars in partition constraint expressions to a separate function map_partition_varattnos() and call it from the appropriate places. Doing it in get_qual_from_partbound(), as is now, would produce wrong result in certain multi-level partitioning cases, because it only considers the current pair of parent-child relations. In certain multi-level partitioning cases, attnums for the same key attribute(s) might differ between various levels causing the same attribute to be numbered differently in different instances of the Var corresponding to a given attribute. With this commit, in generate_partition_qual(), we first generate the the whole partition constraint (considering all levels of partitioning) and then do the mapping, so that Vars in the final expression are numbered according the leaf relation (to which it is supposed to apply). Amit Langote, reviewed by me. http://git.postgresql.org/pg/commitdiff/0563a3a8b59150bf3cc8b2b7077f684e0eaf8aff

Bruce Momjian pushed:

Peter Eisentraut pushed:

Correctifs en attente

Amul Sul sent in two more revisions of a patch to implement pg_background.

Dilip Kumar sent in four more revisions of a patch to implement parallel bitmap heap scan.

Euler Taveira de Oliveira sent in another revision of a patch to move collation import to the backend.

Anastasia Lubennikova and Erik Rijkers traded patches to implement covering + unique indexes.

Takayuki Tsunakawa sent in another revision of a patch to support huge pages on Windows.

Jon Nelson sent in another revision of a patch to guc-ify the formerly hard-coded MAX_SEND_SIZE to max_wal_send.

Ãlvaro Herrera sent in another revision of a patch to help fix pg_dump and COPY bugs around big lines.

David Fetter sent in three more revisions of a patch to add an extension that disallows simple UPDATEs and DELETEs that lack any WHERE clause.

Ãlvaro Herrera sent in a patch to remove floats from the bootstrap scanner and parser.

Antonin Houska sent in two revisions of a patch to implement grouped base relation, which is infrastructure for distributed aggregation.

Vladimir Rusinov sent in four more revisions of a patch to rename things *xlog* to things *wal*.

Keith Fiske and Amit Langote traded patches to fix up declarative partitioning.

Haribabu Kommi sent in another revision of a patch to add macaddr 64 bit (EUI-64) datatype support.

Michaël Paquier sent in two more revisions of a patch to add compression-level adjustment to pg_receivelog.

Kyotaro HORIGUCHI sent in another revision of a patch to implement radix trees for character conversion.

Ashutosh Sharma sent in another revision of a patch to add microvacuum support for hash indexes.

Pavel Stěhule sent in another revision of a patch to implement \gstore and friends in psql.

Matheus de Oliveira sent in another revision of a patch to add ALTER DEFAULT PRIVILEGES with GRANT/REVOKE ON SCHEMAS.

Rafia Sabih sent in two revisions of a patch to enable passing query string to workers.

Jesper Pedersen and Ashutosh Sharma traded patches to add hash index support to the pageinspect contrib extension.

Nikita Glukhovs sent in another revision of a patch to add a recursive version of json_populate_record().

David Rowley sent in a patch to make connection_limit ignore bgworkers.

Peter Eisentraut sent in a patch to factor out the many copies of atooid.

Robert Haas sent in another revision of a patch to implement gather merge.

Etsuro Fujita sent in a patch to rearrange some function declarations for correctness.

Pavel Stěhule sent in a patch to add some checks to PL/pgsql.

Fabien COELHO and Rafia Sabih traded patches to give pgbench a way to use backslash as a continuation character.

Ãlvaro Herrera and Pavel StÄ›hule traded patches to implement xmltable().

Amit Kapila sent in another revision of a patch to add WAL logging for hash indexes.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Etsuro Fujita sent in another revision of a patch to push more FULL JOINs down to FDWs.

Ashutosh Bapat sent in a patch to remove an unused member root in foreign_glob_cxt.

Dmitry Dolgov sent in another revision of a patch to implement generic type subscripts.

Gilles Darold and Karl O. Pinc traded patches to implement pg_current_logfile().

Rafia Sabih sent in another revision of a patch to add parallel index-only scan.

Mithun Cy sent in a patch to fix a typo in a comment in hashsearch.c.

Ants Aasma sent in another revision of a patch to send hot standby feedback on first connect.

Anastasia Lubennikova sent in a patch to implement IF NOT EXISTS option for CREATE SERVER and CREATE USER MAPPING statements.

Amit Khandekar sent in a patch to prevent useless VACUUMs.

Peter Moser sent in another revision of a patch to implement temporal query processing with range types.

Etsuro Fujita sent in another revision of a patch to fix a bug in the Postgres FDW.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Petr Jelínek sent in two more revisions of a patch to implement logical replication.

par chl le jeudi 19 janvier 2017 à 23h47

dimanche 15 janvier 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 8 janvier 2017

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170109204928.GA13529@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • In pgbench logging, avoid assuming that instr_times match Unix timestamps. For aggregated logging, pg_bench supposed that printing the integer part of INSTR_TIME_GET_DOUBLE() would produce a Unix timestamp. That was already broken on Windows, and it's about to get broken on most other platforms as well. As in commit 74baa1e3b, we can remove the entanglement at the price of one extra syscall per transaction; though here it seems more convenient to use time(NULL) instead of gettimeofday(), since we only need integral-second precision. I took the time to do some wordsmithing on the documentation about pgbench's logging features, too. Discussion: https://postgr.es/m/8837.1483216839@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/67a875355e4a6db294e9652af5a883876ddeb4a5
  • Use clock_gettime(), if available, in instr_time measurements. The advantage of clock_gettime() is that the API allows the result to be precise to nanoseconds, not just microseconds as in gettimeofday(). Now that it's routinely possible to do tens of plan node executions in 1us, we really need more precision than gettimeofday() can offer for EXPLAIN ANALYZE to accumulate statistics with. Some research shows that clock_gettime() is available on pretty nearly every modern Unix-ish platform, and as far as I have been able to test, it has about the same execution time as gettimeofday(), so there's no loss in switching over. (By the same token, this doesn't do anything to fix the fact that we really wish clock readings were faster. But there's enough win here to justify changing anyway.) A small side benefit is that on most platforms, we can use CLOCK_MONOTONIC instead of CLOCK_REALTIME and thereby render EXPLAIN impervious to concurrent resets of the system clock. (This means that code must not assume that the contents of struct instr_time have any well-defined interpretation as timestamps, but really that was true before.) Some platforms offer nonstandard clock IDs that might be of interest. This patch knows we should use CLOCK_MONOTONIC_RAW on macOS, because it provides more precision and is faster to read than their CLOCK_MONOTONIC. If there turn out to be many more cases where we need special rules, it might be appropriate to handle the selection of clock ID in configure, but for the moment that doesn't seem worth the trouble. Discussion: https://postgr.es/m/31856.1400021891@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/1d63f7d2d180c8708bc12710254eb7b45823440f
  • Allow SSL configuration to be updated at SIGHUP. It is no longer necessary to restart the server to enable, disable, or reconfigure SSL. Instead, we just create a new SSL_CTX struct (by re-reading all relevant files) whenever we get SIGHUP. Testing shows that this is fast enough that it shouldn't be a problem. In conjunction with that, downgrade the logic that complains about pg_hba.conf "hostssl" lines when SSL isn't active: now that's just a warning condition not an error. An issue that still needs to be addressed is what shall we do with passphrase-protected server keys? As this stands, the server would demand the passphrase again on every SIGHUP, which is certainly impractical. But the case was only barely supported before, so that does not seem a sufficient reason to hold up committing this patch. Andreas Karlsson, reviewed by Michael Banck and Michael Paquier Discussion: https://postgr.es/m/556A6E8A.9030400@proxel.se http://git.postgresql.org/pg/commitdiff/de41869b64d57160f58852eab20a27f248188135
  • Disable prompting for passphrase while (re)loading SSL config files. OpenSSL's default behavior when loading a passphrase-protected key file is to open /dev/tty and demand the password from there. It was kinda sorta okay to allow that to happen at server start, but really that was never workable in standard daemon environments. And it was a complete fail on Windows, where the same thing would happen at every backend launch. Yesterday's commit de41869b6 put the final nail in the coffin by causing that to happen at every SIGHUP; even if you've still got a terminal acting as the server's TTY, having the postmaster freeze until you enter the passphrase again isn't acceptable. Hence, override the default behavior with a callback that returns an empty string, ensuring failure. Change the documentation to say that you can't have a passphrase-protected server key, period. If we can think of a production-grade way of collecting a passphrase from somewhere, we might do that once at server startup and use this callback to feed it to OpenSSL, but it's far from clear that anyone cares enough to invest that much work in the feature. The lack of complaints about the existing fractionally-baked behavior suggests nobody's using it anyway. Discussion: https://postgr.es/m/29982.1483412575@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/1e942c7474d7b5e4bfa04918d2f68d95902f17b3
  • Re-allow SSL passphrase prompt at server start, but not thereafter. Leave OpenSSL's default passphrase collection callback in place during the first call of secure_initialize() in server startup. Although that doesn't work terribly well in daemon contexts, some people feel we should not break it for anyone who was successfully using it before. We still block passphrase demands during SIGHUP, meaning that you can't adjust SSL configuration on-the-fly if you used a passphrase, but this is no worse than what it was before commit de41869b6. And we block passphrase demands during EXEC_BACKEND reloads; that behavior wasn't useful either, but at least now it's documented. Tweak some related log messages for more readability, and avoid issuing essentially duplicate messages about reload failure caused by a passphrase. Discussion: https://postgr.es/m/29982.1483412575@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/6667d9a6d77b9a6eac89638ac363b6d03da253c1
  • Prefer int-wide pg_atomic_flag over char-wide when using gcc intrinsics. configure can only probe the existence of gcc intrinsics, not how well they're implemented, and unfortunately the answer is sometimes "badly". In particular we've found that multiple compilers fail to implement char-width __sync_lock_test_and_set() correctly on PPC; and even a correct implementation would necessarily be pretty inefficient, since that hardware has only a word-wide primitive to work with. Given the knowledge we've accumulated in s_lock.h, it appears that it's best to rely on int-width TAS operations on most non-Intel architectures. Hence, pick int not char when both are nominally available to us in generic-gcc.h (note that that code is not used for x86[_64]). Back-patch to fix regression test failures on FreeBSD/PPC. Ordinarily back-patching a change like this would be verboten because of ABI breakage. But since pg_atomic_flag is not yet used in any Postgres data structure, there's no ABI to break. It seems safer to back-patch to avoid possible gotchas, if someday we do back-patch something that uses pg_atomic_flag. Discussion: https://postgr.es/m/25414.1483076673@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/698127a4a9bc3c74659bf0e5383b1ed99aeb1570
  • Handle OID column inheritance correctly in ALTER TABLE ... INHERIT. Inheritance operations must treat the OID column, if any, much like regular user columns. But MergeAttributesIntoExisting() neglected to do that, leading to weird results after a table with OIDs is associated to a parent with OIDs via ALTER TABLE ... INHERIT. Report and patch by Amit Langote, reviewed by Ashutosh Bapat, some adjustments by me. It's been broken all along, so back-patch to all supported branches. Discussion: https://postgr.es/m/cb13cfe7-a48c-5720-c383-bb843ab28298@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/d86f40009b6b019f794819a9af9038cff0cac6f3
  • Fix handling of empty arrays in array_fill(). array_fill(..., array[0]) produced an empty array, which is probably what users expect, but it was a one-dimensional zero-length array which is not our standard representation of empty arrays. Also, for no very good reason, it rejected empty input arrays; that case should be allowed and produce an empty output array. In passing, remove the restriction that the input array(s) have lower bound 1. That seems rather pointless, and it would have needed extra complexity to make the check deal with empty input arrays. Per bug #14487 from Andrew Gierth. It's been broken all along, so back-patch to all supported branches. Discussion: https://postgr.es/m/20170105152156.10135.64195@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/82f8107b92c9104ec9d9465f3f6a4c6dab4c124a
  • Invalidate cached plans on FDW option changes. This fixes problems where a plan must change but fails to do so, as seen in a bug report from Rajkumar Raghuwanshi. For ALTER FOREIGN TABLE OPTIONS, do this through the standard method of forcing a relcache flush on the table. For ALTER FOREIGN DATA WRAPPER and ALTER SERVER, just flush the whole plan cache on any change in pg_foreign_data_wrapper or pg_foreign_server. That matches the way we handle some other low-probability cases such as opclass changes, and it's unclear that the case arises often enough to be worth working harder. Besides, that gives a patch that is simple enough to back-patch with confidence. Back-patch to 9.3. In principle we could apply the code change to 9.2 as well, but (a) we lack postgres_fdw to test it with, (b) it's doubtful that anyone is doing anything exciting enough with FDWs that far back to need this desperately, and (c) the patch doesn't apply cleanly. Patch originally by Amit Langote, reviewed by Etsuro Fujita and Ashutosh Bapat, who each contributed substantial changes as well. Discussion: https://postgr.es/m/CAKcux6m5cA6rRPTKkqVdJ-R=KKDfe35Q_ZuUqxDSV_4hwga=og@mail.gmail.com http://git.postgresql.org/pg/commitdiff/c52d37c8b3674f1ca9ead297480ade0ac9c54174
  • Merge two copies of tuple-building code in pltcl.c. Make pltcl_trigger_handler() construct modified tuples using pltcl_build_tuple_result(), rather than its own copy of essentially the same logic. This results in slightly different message wording for the error cases, and in one case a different SQLSTATE, but it seems unlikely that any existing applications are depending on any of those details. While at it, fix a typo in commit 26abb50c4: pltcl_build_tuple_result was applying encoding conversion in the wrong direction. That would be a back-patchable bug fix, except the code hasn't shipped yet. Jim Nasby, reviewed by me Discussion: https://postgr.es/m/d2c6425a-d9e0-f034-f774-4a872c234d89@BlueTreble.com http://git.postgresql.org/pg/commitdiff/de5fed0d0c704e3d1a928bf420663319d898cee4
  • Get rid of ParseState.p_value_substitute; use a columnref hook instead. I noticed that p_value_substitute, which is a single-purpose kluge I added in 2002 (commit b0422b215), could be replaced by having domainAddConstraint install a parser hook that looks for the name "value". The parser hook code only dates back to 2009, so it's not surprising that we had to kluge this in 2002, but we can do it more cleanly now. http://git.postgresql.org/pg/commitdiff/7c3abe3c92fd3a14a70bc2f888f936cd6fe28c0f
  • Improve documentation of struct ParseState. I got annoyed about how some fields of ParseState were documented in the struct's block comment and some weren't; not all of the latter are trivial. Fix that. Also reorder a couple of fields that seem to have been placed rather randomly, or maybe with an idea of avoiding padding space; but there are never so many ParseStates in existence at one time that we ought to value pad space over readability. http://git.postgresql.org/pg/commitdiff/3c40594e6eeabb3a8ad22aee93de3a19c41efdc2

Heikki Linnakangas pushed:

Peter Eisentraut pushed:

Bruce Momjian pushed:

Magnus Hagander pushed:

  • Make wal streaming the default mode for pg_basebackup. Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs http://git.postgresql.org/pg/commitdiff/9a4d51077c96c10322582211781bb969b51822ff
  • Attempt to handle pending-delete files on Windows. These files are deleted but not yet gone from the filesystem. Operations on them will return ERROR_DELETE_PENDING. With this we start treating that as ENOENT, meaning files does not exist (which is the state it will soon reach). This should be safe in every case except when we try to recreate a file with exactly the same name. This is an operation that PostgreSQL does very seldom, so hopefully that won't happen much -- and even if it does, this treatment should be no worse than treating it as an unhandled error. We've been un able to reproduce the bug reliably, so pushing this to master to get buildfarm coverage and other testing. Once it's proven to be stable, it should be considered for backpatching. Discussion: https://postgr.es/m/20160712083220.1426.58667%40wrigleys.postgresql.org Patch by me and Michael Paquier http://git.postgresql.org/pg/commitdiff/9951741bbeb3ec37ca50e9ce3df1808c931ff6a6

Simon Riggs pushed:

Robert Haas pushed:

Stephen Frost pushed:

  • Protect against NULL-dereference in pg_dump. findTableByOid() is allowed to return NULL and we should therefore be checking for that case. getOwnedSeqs() and dumpSequence() shouldn't ever actually see this happen, but given odd circumstances it might and commit f9e439b1 probably shouldn't have removed that check. Pointed out by Coverity. Initial patch from Michael Paquier. Back-patch to 9.6, where that commit had removed the check. http://git.postgresql.org/pg/commitdiff/d74ecbc8d85eb7a2aa1d5516c5c38d6ab0cbbd82
  • Add basic pg_dumpall/pg_restore TAP tests. For reasons unknown, pg_dumpall and pg_restore managed to escape the basic set of TAP tests that were added for pg_dump in 6bd356c3, so let's get them added now. A few minor adjustments are also made to the dump/restore tests to improve code coverage for pg_restore/pg_dumpall. http://git.postgresql.org/pg/commitdiff/9b815a8ff227e62442259e0fbabc5cf37e433df9

Correctifs en attente

Thomas Munro sent in a patch to add an isolation test for SERIALIZABLE READ ONLY DEFERRABLE.

Ashutosh Bapat sent in another revision of a patch to support partition-wise join between multi-level partitioned tables.

Thomas Munro sent in another revision of a patch to implement causal reads.

Pavan Deolasee sent in two more revisions of a patch to implement WARM.

Ashutosh Bapat sent in a patch to cast nodes more safely.

Thomas Munro sent in another revision of a patch to help measure replay lag.

KaiGai Kohei sent in another revision of a patch to add PassDownLimitBound for ForeignScan/CustomScan.

Mithun Cy sent in two more revisions of a patch to cache hash index meta page.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Peter Geoghegan sent in another revision of a patch to implement parallel tuplesort.

Tom Lane sent in a patch to expand the existing API to allow the AM to return either a heap or index tuple.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Heikki Linnakangas sent in a patch to replace isMD5() with a more future-proof way to check if password is encrypted and use "plain:" prefix for plaintext passwords stored in pg_authid.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Craig Ringer sent in two more revisions of a patch to implement logical decoding on standbys.

Peter Eisentraut sent in two revisions of a patch to fix some infelicities in the logical replication protocol and output plugin.

Peter Eisentraut sent in another revision of a patch to add logical replication workers.

Peter Eisentraut and Petr Jelínek traded patches to add PUBLICATION catalogs and DDL.

Ãlvaro Herrera sent in three revisions of a patch to fix and error in ALTER TABLE ... ALTER TYPE.

Haribabu Kommi sent in another revision of a patch to implement a pg_hba_file_settings view and add TAP tests for same.

David Fetter sent in two more revisions of a patch to disable simple UPDATEs and DELETEs which lack any WHERE clause.

Peter Eisentraut sent in another revision of a patch to fix some infelicities between pg_basebackups and slots.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Kuntal Ghosh sent in another revision of a patch to implement a WAL consistency check facility.

Craig Ringer sent in another revision of a patch to add PostgresNode methods to wait for node catchup and expand streaming replication tests to cover hot standby feedback and physical replication slots.

Michaël Paquier sent in another revision of a patch to bring the TAP test docs for PostgresNode up to date.

Peter Eisentraut sent in another revision of a patch to generate fmgr prototypes automatically.

Masahiko Sawada sent in a patch to add a GUC for cleanup indexes threshold.

Ashutosh Bapat sent in a patch to add an option to EXPLAIN: SUMMARY option which behaves as: SUMMARY when ON prints planning time, and prints planning time in EXPLAIN EXECUTE output.

Takayuki Tsunakawa sent in two more revisions of a patch to support huge pages on Windows.

Haribabu Kommi sent in another revision of a patch to add macaddr 64 bit (EUI-64) datatype support.

Ashutosh Sharma sent in two more revisions of a patch to add microvacuum support for hash indexes.

Jesper Pedersen sent in another revision of a patch to add support for hash index in pageinspect contrib module.

Tatsuo Ishii sent in a patch to fix a mistaken tag in the documentation for user name maps.

Vitaly Burovoy sent in another revision of a patch to add check for overflow to 'interval' functions.

Andrew Gierth sent in a patch to add hash support for grouping sets.

Jim Nasby sent in another revision of a patch to implement faster methods for getting SPI results in PL/PythonU.

Vladimir Rusinov sent in another revision of a patch to rename things xlog to things wal.

Jonathon Nelson sent in a patch to guc-ify the formerly hard-coded MAX_SEND_SIZE to max_wal_send.

Takeshi Ideriha sent in another revision of a patch to add DECLARE STATEMENT setting up a connection in ECPG.

Amit Langote sent in a patch to fix up some infelicities in declarative partitioning.

Ashutosh Sharma sent in another revision of a patch to add pgstathashindex() to get hash index table statistics.

Thomas Munro sent in another revision of a patch to implement barriers.

Thomas Munro sent in another revision of a patch to add parallel shared hash.

Peter Geoghegan sent in a patch to fix a subtle bug in "Simplify tape block format" commit.

Masahiko Sawada sent in another revision of a patch to enable block level parallel vacuum.

Michaël Paquier sent in another revision of a patch to add compression levels to pg_receivexlog.

Amul Sul sent in two more revisions of a patch to add a pg_background contrib module.

Pavel Stěhule sent in a patch to add pragmas to PL/pgsql.

Fabien COELHO sent in another revision of a patch to enable pgbench to store select results into variables.

Tom Lane sent in a patch to disallow output columns of type UNKNOWN.

Ryan Murphy sent in a patch to show type name and constraints for errors in inherited tables.

Jim Nasby sent in two more revisions of a patch to add more test coverage for PL/Tcl.

Victor Wagner sent in two revisions of a patch to add explicit subtransactions to PL/Tcl.

Haribabu Kommi sent in a patch to fix a bug in the patch to add a pg_hba_settings view.

par chl le dimanche 15 janvier 2017 à 11h46

samedi 7 janvier 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 1er janvier 2017

Les nouveautés des produits dérivés

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170101214521.GA22193@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Remove triggerable Assert in hashname(). hashname() asserted that the key string it is given is shorter than NAMEDATALEN. That should surely always be true if the input is in fact a regular value of type "name". However, for reasons of coding convenience, we allow plain old C strings to be treated as "name" values in many places. Some SQL functions accept arbitrary "text" inputs, convert them to C strings, and pass them otherwise-untransformed to syscache lookups for name columns, allowing an overlength input value to trigger hashname's Assert. This would be a DOS problem, except that it only happens in assert-enabled builds which aren't recommended for production. In a production build, you'll just get a name lookup error, since regardless of the hash value computed by hashname, the later equality comparison checks can't match. Likewise, if the catalog lookup is done by seqscan or indexscan searches, there will just be a lookup error, since the name comparison functions don't contain any similar length checks, and will see an overlength input as unequal to any stored entry. After discussion we concluded that we should simply remove this Assert. It's inessential to hashname's own functionality, and having such an assertion in only some paths for name lookup is more of a foot-gun than a useful check. There may or may not be a case for the affected callers to do something other than let the name lookup fail, but we'll consider that separately; in any case we probably don't want to change such behavior in the back branches. Per report from Tushar Ahuja. Back-patch to all supported branches. Report: https://postgr.es/m/7d0809ee-6f25-c9d6-8e74-5b2967830d49@enterprisedb.com Discussion: https://postgr.es/m/17691.1482523168@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/54386f3578258caa5a1de97c434eee2c9ee2ab06
  • Fix interval_transform so it doesn't throw away non-no-op casts. interval_transform() contained two separate bugs that caused it to sometimes mistakenly decide that a cast from interval to restricted interval is a no-op and throw it away. First, it was wrong to rely on dt.h's field type macros to have an ordering consistent with the field's significance; in one case they do not. This led to mistakenly treating YEAR as less significant than MONTH, so that a cast from INTERVAL MONTH to INTERVAL YEAR was incorrectly discarded. Second, fls(1<<k) produces k+1 not k, so comparing its output directly to SECOND was wrong. This led to supposing that a cast to INTERVAL MINUTE was really a cast to INTERVAL SECOND and so could be discarded. To fix, get rid of the use of fls(), and make a function based on intervaltypmodout to produce a field ID code adapted to the need here. Per bug #14479 from Piotr Stefaniak. Back-patch to 9.2 where transform functions were introduced, because this code was born broken. Discussion: https://postgr.es/m/20161227172307.10135.7747@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/f0774abde868e0b5a2acbe75b5028884752f739d
  • Remove manual breaks in NodeTag assignments to fix duplicate tag numbers. Commit f0e44751d added new node tags at a place in the tag numbering where there was no daylight left before the next hard-coded number, resulting in some duplicate tag assignments. This doesn't seem to have caused any big problem so far, but it's surely trouble waiting to happen. We could adjust the manually assigned breakpoints to make more room, but that just leaves the same hazard waiting to strike again in future. What seems like a better idea is to get rid of the manual assignments and leave NodeTags to be automatically assigned, consecutively from one on up. This means that any change in the tag list forces a backend-wide recompile, but realistically that's usually needed anyway. Discussion: https://postgr.es/m/29670.1482942811@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/80a7298b9eb7f108ef20be6ee00d9513a43c61a0
  • Fix unstable regression test results. Commit 2ac3ef7a0 added a query with an underdetermined output row order; it has failed multiple times in the buildfarm since then. Add an ORDER BY to fix. Also, don't rely on a DROP CASCADE to drop in a well-determined order; that hasn't failed yet but I don't trust it much, and we're not saving any typing by using CASCADE anyway. http://git.postgresql.org/pg/commitdiff/257d8157205a7be5f9799e8941b922521d678a25
  • Avoid assuming that instr_time == struct timeval in pgbench logging. This code was presuming undue familiarity with the contents of the instr_time struct. That was already broken on Windows, and it's about to get broken on most other platforms as well. The simplest solution that preserves the current output definition is to just do our own gettimeofday() call here. Realistically, the extra cost is probably negligible in comparison to everything else that's going on in a pgbench transaction, so it's not worth sweating over. On Windows, the precision delivered by gettimeofday() is lower than one could wish, but this is still a big improvement over printing zeroes, as the code did before. Discussion: https://postgr.es/m/8837.1483216839@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/74baa1e3b89c1651ade1afeffc715cac24041e2f

Magnus Hagander pushed:

Andrew Dunstan pushed:

Peter Eisentraut pushed:

Correctifs en attente

Pavan Deolasee sent in another revision of a patch to fix some issues in WARM.

Kyotaro HORIGUCHI sent in another revision of a patch to clean up negative cache of pg_statistic when dropping a relation and clean up negative cache of pg_class when dropping a schema.

Kyotaro HORIGUCHI sent in another revision of a patch to redesign psql's tab completion and use that infrastructure to implement IF (NOT) EXISTS completion.

Ashutosh Bapat sent in another revision of a patch to imlement partition-wise join for join between (declaratively) partitioned tables.

Amit Kapila sent in another revision of a patch to implement WAL for hash indexes.

Dmitry Dolgov sent in another revision of a patch to implement generic type subscripting.

Amit Langote sent in five more revisions of a patch to implement declarative partitioning.

Amit Langote sent in a patch to fix an infelicity between OID columns and table inheritance.

Mithun Cy sent in two more revisions of a patch to cache hash index meta pages.

Etsuro Fujita sent in another revision of a patch to push more FULL joins to the PostgreSQL FDW.

Amit Kapila sent in another revision of a patch to implement parallel index scans.

Nikita Glukhov sent in another revision of a patch to implement recursive json_populate_record().

Peter Eisentraut sent in another revision of a patch to add ICU support.

Rafia Sabih sent in another revision of a patch to implement parallel index-only scans.

Amit Kapila sent in a patch to parallelize queries containing subplans.

Thomas Munro sent in a patch to fix dsa tranche registration.

Michaël Paquier sent in two more revisions of a patch to enable pg_receivelog to enable adjusting compression for tar format.

Peter Eisentraut sent in a patch to add some information about the systemd RemoveIPC issue to the documentation.

Ideriha Takeshi sent in a patch to add a DECLARE STATEMENT command in ECPG.

Craig Ringer sent in another revision of a patch to introduce txid_status(bigint) to get status of an xact.

Thomas Munro sent in another revision of a patch to allow measuring replay lag.

Claudio Freire sent in another revision of a patch to enable VACUUM to use more than 1GB of work mem.

Craig Ringer sent in two revisions of a patch to fix a minor race between commit_ts slru truncation and lookups.

Tom Lane sent in another revision of a patch to improve RLS planning.

Ãlvaro Herrera sent in two revisions of a patch to rewrite HeapSatisfiesHOTAndKey.

Ãlvaro Herrera sent in two more revisions of a patch to implement indirect indexes.

Vik Fearing sent in a patch to change the encoding of the French translation encoding from ISO-8859-1 to UTF-8.

Jim Nasby sent in another revision of a patch to increase pltcl test coverage.

Peter Eisentraut sent in another revision of a patch to add background sessions.

Tom Lane sent in a patch to eliminate gettimeofday() on platforms where better options are available.

Haribabu Kommi sent in a patch to implement a columnar storage extension.

Fabien COELHO sent in another revision of a patch to fix some pg_stat_statements query normalization issues with combined queries.

Yury Zhuravlev sent in another revision of a patch to implement a CMake-based build system.

Petr Jelínek sent in another revision of a patch to implement logical replication.

Michaël Paquier sent in another revision of a patch to fix a potential data loss of 2PC files.

Fabrízio de Royes Mello sent in another revision of a patch to implement COMMENT ON CURRENT DATABASE.

Amit Kapila sent in another revision of a patch to speed up clog Access by increasing CLOG buffers.

Amit Kapila sent in a patch to fix an issue where group clear xid can leak semaphore count.

Peter Eisentraut sent in a patch to generate fmgr prototypes automatically.

Peter Eisentraut sent in another revision of a patch to implement sequence data types.

Peter Eisentraut sent in a patch to put "use strict" in all Perl programs.

Peter Eisentraut sent in a patch to create the INSTALL file via XSLT and Pandoc. This is one more step on the way to replacing the old tool chain of jade and lynx.

Peter Eisentraut sent in a patch to automatically casts the result of copyNode() back to the input type, if the compiler supports something like typeof().

Stas Kelvich sent in another revision of a patch to implement logical decoding of two-phase transactions.

Magnus Hagander sent in another revision of a patch to support huge pages on Windows.

Peter Eisentraut sent in another revision of a patch to add identity columns per the SQL standard.

Magnus Hagander sent in a patch to change the backup and replication defaults to something more useful.

Peter Eisentraut sent in another revision of a patch to allow DROP FUNCTION to drop multiple functions in one command.

Peter Eisentraut sent in a patch to implement safer node casting.

Magnus Hagander sent in another revision of a patch to make streaming the pg_basebackup default.

David Fetter sent in another revision of a patch to add a hook which allows disabling simple UPDATEs and DELETEs without a WHERE clause.

Thomas Munro sent in a patch to add an isolation test for SERIALIZABLE READ ONLY DEFERRABLE.

Pavel Stěhule sent in a patch to add some runtime checks to PL/pgsql.

Simon Riggs sent in another revision of a patch to make some changes to the recovery.conf API.

par chl le samedi 7 janvier 2017 à 13h46

Nouvelles hebdomadaires de PostgreSQL - 25 décembre 2016

Le pgDay Paris 2017 aura lieu à Paris (France) le 23 mars 2017. L'appel à conférenciers court jusqu'au 9 janvier : http://2017.pgday.paris/callforpapers/

Le PGDay nordique se tiendra à Stockholm (Suède) au Sheraton Hotel, le 21 mars 2017. L'appel à conférenciers expire le 9 janvier 2017 : https://2017.nordicpgday.org/cfp/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/
  • PgConf.Russia 2017 se déroulera du 15 au 17 mars 2017 à Moscou : https://pgconf.ru/en
  • Le PGDay Asia 2017 s'étendra du 17 au 18 mars à Singapour. L'appel à conférenciers s'éteindra le 16 janvier 2017 : http://tinyurl.com/pgDay-Asia-2017-Cfp
  • PGCon 2017 aura lieu à Ottawa du 23 au 26 mai. Les propositions seront attendues jusqu'au 19 janvier 2017 : http://www.pgcon.org/2017/papers.php

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161225225258.GA21777@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Magnus Hagander pushed:

Fujii Masao pushed:

  • Support quorum-based synchronous replication. This feature is also known as "quorum commit" especially in discussion on pgsql-hackers. This commit adds the following new syntaxes into synchronous_standby_names GUC. By using FIRST and ANY keywords, users can specify the method to choose synchronous standbys from the listed servers. FIRST num_sync (standby_name [, ...]) ANY num_sync (standby_name [, ...]) The keyword FIRST specifies a priority-based synchronous replication which was available also in 9.6 or before. This method makes transaction commits wait until their WAL records are replicated to num_sync synchronous standbys chosen based on their priorities. The keyword ANY specifies a quorum-based synchronous replication and makes transaction commits wait until their WAL records are replicated to *at least* num_sync listed standbys. In this method, the values of sync_state.pg_stat_replication for the listed standbys are reported as "quorum". The priority is still assigned to each standby, but not used in this method. The existing syntaxes having neither FIRST nor ANY keyword are still supported. They are the same as new syntax with FIRST keyword, i.e., a priorirty-based synchronous replication. Author: Masahiko Sawada Reviewed-By: Michael Paquier, Amit Kapila and me Discussion: <CAD21AoAACi9NeC_ecm+Vahm+MMA6nYh=Kqs3KB3np+MBOS_gZg@mail.gmail.com> Many thanks to the various individuals who were involved in discussing and developing this feature. http://git.postgresql.org/pg/commitdiff/3901fd70cc7ccacef1b0549a6835bb7d8dcaae43
  • Forbid invalid combination of options in pg_basebackup. Commit 56c7d8d4552180fd66fe48423bb2a9bb767c2d87 allowed pg_basebackup to stream WAL in tar mode. But there is the restriction that WAL streaming in tar mode works only when the value - (dash) is not specified as output directory. This means that the combination of three options "-D -", "-F t" and "-X stream" is invalid. However, previously, even when those options were specified at the same time, pg_basebackup background process unexpectedly started streaming WAL. And then it exited with an error. This commit changes pg_basebackup so that it errors out on such invalid combination of options at the beginning. Reviewed by Magnus Hagander, and patch by me. http://git.postgresql.org/pg/commitdiff/ecbdc4c555f43b1ac284c734752b00c2ea6f277b

Robert Haas pushed:

Tom Lane pushed:

  • Fix handling of phrase operator removal while removing tsquery stopwords. The distance of a removed phrase operator should propagate up to a parent phrase operator if there is one, but this only worked correctly in left-deep trees. Throwing in a few parentheses confused it completely, as indeed was illustrated by bizarre results in existing regression test cases. To fix, track unaccounted-for distances that should propagate to the left and to the right of the current node, rather than trying to make it work with only one returned distance. Also make some adjustments to behave as well as we can for cases of intermixed phrase and regular (AND/OR) operators. I don't think it's possible to be 100% correct for that without a rethinking of the tsquery representation; for example, maybe we should just not drop stopword nodes at all underneath phrase operators. But this is better than it was, and changing tsquery representation wouldn't be safely back-patchable. While at it, I simplified the API of the clean_fakeval_intree function a bit by getting rid of the "char *result" output parameter; that wasn't doing anything that wasn't redundant with whether the result node is NULL or not, and testing for NULL seems a lot clearer/safer. This is part of a larger project to fix various infelicities in the phrase-search implementation, but this part seems comittable on its own. Back-patch to 9.6 where phrase operators were introduced. Discussion: https://postgr.es/m/28215.1481999808@sss.pgh.pa.us Discussion: https://postgr.es/m/26706.1482087250@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/2604438472c897fbbd1568b1a8ee177ba8cdb6e3
  • Fix minor error message style violation. Primary error messages should not end with a period, since they're generally not written as full sentences. Oversight in 41493bac3. http://git.postgresql.org/pg/commitdiff/7d41a2bd3eef4de64ae8f6f683457f12f9407c5d
  • Fix minor oversights in nodeAgg.c. aggstate->evalproj is always set up by ExecInitAgg, so there's no need to test. Doing so led Coverity to think that we might be intending "slot" to be possibly NULL here, and it quite properly complained that the rest of combine_aggregates() wasn't prepared for that. Also fix a couple of obvious thinkos in Asserts checking that "inputoff" isn't past the end of the slot. Errors introduced in commit 8ed3f11bb, so no need for back-patch. http://git.postgresql.org/pg/commitdiff/c080b223a7a3991524a5287416a0ad756c15a098
  • Fix strange behavior (and possible crashes) in full text phrase search. In an attempt to simplify the tsquery matching engine, the original phrase search patch invented rewrite rules that would rearrange a tsquery so that no AND/OR/NOT operator appeared below a PHRASE operator. But this approach had numerous problems. The rearrangement step was missed by ts_rewrite (and perhaps other places), allowing tsqueries to be created that would cause Assert failures or perhaps crashes at execution, as reported by Andreas Seltenreich. The rewrite rules effectively defined semantics for operators underneath PHRASE that were buggy, or at least unintuitive. And because rewriting was done in tsqueryin() rather than at execution, the rearrangement was user-visible, which is not very desirable --- for example, it might cause unexpected matches or failures to match in ts_rewrite. As a somewhat independent problem, the behavior of nested PHRASE operators was only sane for left-deep trees; queries like "x <-> (y <-> z)" did not behave intuitively at all. To fix, get rid of the rewrite logic altogether, and instead teach the tsquery execution engine to manage AND/OR/NOT below a PHRASE operator by explicitly computing the match location(s) and match widths for these operators. This requires introducing some additional fields into the publicly visible ExecPhraseData struct; but since there's no way for third-party code to pass such a struct to TS_phrase_execute, it shouldn't create an ABI problem as long as we don't move the offsets of the existing fields. Another related problem was that index searches supposed that "!x <-> y" could be lossily approximated as "!x & y", which isn't correct because the latter will reject, say, "x q y" which the query itself accepts. This required some tweaking in TS_execute_ternary along with the main tsquery engine. Back-patch to 9.6 where phrase operators were introduced. While this could be argued to change behavior more than we'd like in a stable branch, we have to do something about the crash hazards and index-vs-seqscan inconsistency, and it doesn't seem desirable to let the unintuitive behaviors induced by the rewriting implementation stand as precedent. Discussion: https://postgr.es/m/28215.1481999808@sss.pgh.pa.us Discussion: https://postgr.es/m/26706.1482087250@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/89fcea1ace40bc025beea2758a80bcd56a319a6f
  • Fix detection of unfinished Unicode surrogate pair at end of string. The U&'...' and U&"..." syntaxes silently discarded a surrogate pair start (that is, a code between U+D800 and U+DBFF) if it occurred at the very end of the string. This seems like an obvious oversight, since we throw an error for every other invalid combination of surrogate characters, including the very same situation in E'...' syntax. This has been wrong since the pair processing was added (in 9.0), so back-patch to all supported branches. Discussion: https://postgr.es/m/19113.1482337898@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/a8ae12322ae056ebbe9d83cf16b4c92e86a6ac28
  • Give a useful error message if uuid-ossp is built without preconfiguration. Before commit b8cc8f947, it was possible to build contrib/uuid-ossp without having told configure you meant to; you could just cd into that directory and "make". That no longer works because the code depends on configure to have done header and library probes, but the ensuing error messages are not so easy to interpret if you're not an old C hand. We've gotten a couple of complaints recently from people trying to do this the low-tech way, so add an explicit #error directing the user to use --with-uuid. (In principle we might want to do something similar in the other optionally-built contrib modules; but I don't think any of the others have ever worked without preconfiguration, so there are no bad habits to break people of.) Back-patch to 9.4 where the previous commit came in. Report: https://postgr.es/m/CAHeEsBf42AWTnk=1qJvFv+mYgRFm07Knsfuc86Ono8nRjf3tvQ@mail.gmail.com Report: https://postgr.es/m/CAKYdkBrUaZX+F6KpmzoHqMtiUqCtAW_w6Dgvr6F0WTiopuGxow@mail.gmail.com http://git.postgresql.org/pg/commitdiff/b86515da1a73d0a2e23aca728f18b5f9e809e89f
  • Fix handling of expanded objects in CoerceToDomain and CASE execution. When the input value to a CoerceToDomain expression node is a read-write expanded datum, we should pass a read-only pointer to any domain CHECK expressions and then return the original read-write pointer as the expression result. Previously we were blindly passing the same pointer to all the consumers of the value, making it possible for a function in CHECK to modify or even delete the expanded value. (Since a plpgsql function will absorb a passed-in read-write expanded array as a local variable value, it will in fact delete the value on exit.) A similar hazard of passing the same read-write pointer to multiple consumers exists in domain_check() and in ExecEvalCase, so fix those too. The fix requires adding MakeExpandedObjectReadOnly calls at the appropriate places, which is simple enough except that we need to get the data type's typlen from somewhere. For the domain cases, solve this by redefining DomainConstraintRef.tcache as okay for callers to access; there wasn't any reason for the original convention against that, other than not wanting the API of typcache.c to be any wider than it had to be. For CASE, there's no good solution except to add a syscache lookup during executor start. Per bug #14472 from Marcos Castedo. Back-patch to 9.5 where expanded values were introduced. Discussion: https://postgr.es/m/15225.1482431619@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/cd1b215692d2cadf499936dba05f1f60bce59344
  • Fix CREATE TABLE ... LIKE ... WITH OIDS. Having a WITH OIDS specification should result in the creation of an OID column, but commit b943f502b broke that in the case that there were LIKE tables without OIDS. Commentary in that patch makes it look like this was intentional, but if so it was based on a faulty reading of what inheritance does: the parent tables can add an OID column, but they can't subtract one. AFAICS, the behavior ought to be that you get an OID column if any of the inherited tables, LIKE tables, or WITH clause ask for one. Also, revert that patch's unnecessary split of transformCreateStmt's loop over the tableElts list into two passes. That seems to have been based on a misunderstanding as well: we already have two-pass processing here, we don't need three passes. Per bug #14474 from Jeff Dafoe. Back-patch to 9.6 where the misbehavior was introduced. Report: https://postgr.es/m/20161222145304.25620.47445@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/1ead0208b2178089b024caa2d1465a3f3056a45c
  • Spellcheck: s/descendent/descendant/g. I got a little annoyed by reading documentation paragraphs containing both spellings within a few lines of each other. My dictionary says "descendant" is the preferred spelling, and it's certainly the majority usage in our tree, so standardize on that. For one usage in parallel.sgml, I thought it better to rewrite to avoid the term altogether. http://git.postgresql.org/pg/commitdiff/ff33d1456ea098e160cbbc74b332656c06abc2ab
  • Doc: improve index entry for "median". We had an index entry for "median" attached to the percentile_cont function entry, which was pretty useless because a person following the link would never realize that that function was the one they were being hinted to use. Instead, make the index entry point at the example in syntax-aggregates, and add a <seealso> link to "percentile". Also, since that example explicitly claims to be calculating the median, make it use percentile_cont not percentile_disc. This makes no difference in terms of the larger goals of that section, but so far as I can find, nearly everyone thinks that "median" means the continuous not discrete calculation. Per gripe from Steven Winfield. Back-patch to 9.4 where we introduced percentile_cont. Discussion: https://postgr.es/m/20161223102056.25614.1166@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/3c9d398484fb6e188e665be8299d6e5e89924c94
  • Replace enum InhOption with simple boolean. Now that it has only INH_NO and INH_YES values, it's just weird that it's not a plain bool, so make it that way. Also rename RangeVar.inhOpt to "inh", to be like RangeTblEntry.inh. My recollection is that we gave it a different name specifically because it had a different representation than the derived bool value, but it no longer does. And this is a good forcing function to be sure we catch any places that are affected by the change. Bump catversion because of possible effect on stored RangeVar nodes. I'm not exactly convinced that we ever store RangeVar on disk, but we have a readfuncs function for it, so be cautious. (If we do do so, then commit e13486eba was in error not to bump catversion.) Follow-on to commit e13486eba. Discussion: http://postgr.es/m/CA+TgmoYe+EG7LdYX6pkcNxr4ygkP4+A=jm9o-CPXyOvRiCNwaQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/fe591f8bf68db9bf81f278acce6239ee68cd4ed6
  • Fix incorrect error reporting for duplicate data in \crosstabview. \crosstabview's complaint about multiple entries for the same crosstab cell quoted the wrong row and/or column values. It would accidentally appear to work if the data had been in strcmp() order to start with, which probably explains how we missed noticing this during development. This could be fixed in more than one way, but the way I chose was to hang onto both result pointers from bsearch() and use those to get at the value names. In passing, avoid casting away const in the bsearch comparison functions. No bug there, just poor style. Per bug #14476 from Tomonari Katsumata. Back-patch to 9.6 where \crosstabview was introduced. Report: https://postgr.es/m/20161225021519.10139.45460@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/a3aef88e6a9c5822eb4a5ad0744b15dc6e8a5d86

Heikki Linnakangas pushed:

Peter Eisentraut pushed:

Dean Rasheed pushed:

  • Fix order of operations in CREATE OR REPLACE VIEW. When CREATE OR REPLACE VIEW acts on an existing view, don't update the view options until after the view query has been updated. This is necessary in the case where CREATE OR REPLACE VIEW is used on an existing view that is not updatable, and the new view is updatable and specifies the WITH CHECK OPTION. In this case, attempting to apply the new options to the view before updating its query fails, because the options are applied using the ALTER TABLE infrastructure which checks that WITH CHECK OPTION is only applied to an updatable view. If new columns are being added to the view, that is also done using the ALTER TABLE infrastructure, but it is important that that still be done before updating the view query, because the rules system checks that the query columns match those on the view relation. Added a comment to explain that, in case someone is tempted to move that to where the view options are now being set. Back-patch to 9.4 where WITH CHECK OPTION was added. Report: https://postgr.es/m/CAEZATCUp%3Dz%3Ds4SzZjr14bfct_bdJNwMPi-gFi3Xc5k1ntbsAgQ%40mail.gmail.com http://git.postgresql.org/pg/commitdiff/58b1362642d47bd7a7ed1157035a38671555e860

Stephen Frost pushed:

  • Fix dumping of casts and transforms using built-in functions. In pg_dump.c dumpCast() and dumpTransform(), we would happily ignore the cast or transform if it happened to use a built-in function because we weren't including the information about built-in functions when querying pg_proc from getFuncs(). Modify the query in getFuncs() to also gather information about functions which are used by user-defined casts and transforms (where "user-defined" means "has an OID >= FirstNormalObjectId"). This also adds to the TAP regression tests for 9.6 and master to cover these types of objects. Back-patch all the way for casts, back to 9.5 for transforms. Discussion: https://www.postgresql.org/message-id/flat/20160504183952.GE10850%40tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/2259bf672cb45b4104dcb835354beeb1c6105b0e
  • For 8.0 servers, get last built-in oid from pg_database. We didn't start ensuring that all built-in objects had OIDs less than 16384 until 8.1, so for 8.0 servers we still need to query the value out of pg_database. We need this, in particular, to distinguish which casts were built-in and which were user-defined. For HEAD, we only worry about going back to 8.0, for the back-branches, we also ensure that 7.0-7.4 work. Discussion: https://www.postgresql.org/message-id/flat/20160504183952.GE10850%40tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/19990918d3fa2a445561627ed415b5891602f7fe
  • Improve ALTER TABLE documentation. The ALTER TABLE documentation wasn't terribly clear when it came to which commands could be combined together and what it meant when they were. In particular, SET TABLESPACE *can* be combined with other commands, when it's operating against a single table, but not when multiple tables are being moved with ALL IN TABLESPACE. Further, the actions are applied together but not really in 'parallel', at least today. Pointed out by: Amit Langote. Improved wording from Tom. Back-patch to 9.4, where the ALL IN TABLESPACE option was added. Discussion: https://www.postgresql.org/message-id/14c535b4-13ef-0590-1b98-76af355a0763%40lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/2d1018ca56f5ddaf0bfb5b4d0133283f3e823301
  • Use TSConfigRelationId in AlterTSConfiguration(). When we are altering a text search configuration, we are getting the tuple from pg_ts_config and using its OID, so use TSConfigRelationId when invoking any post-alter hooks and setting the object address. Further, in the functions called from AlterTSConfiguration(), we're saving information about the command via EventTriggerCollectAlterTSConfig(), so we should be setting commandCollected to true. Also add a regression test to test_ddl_deparse for ALTER TEXT SEARCH CONFIGURATION. Author: Artur Zakirov, a few additional comments by me Discussion: https://www.postgresql.org/message-id/57a71eba-f2c7-e7fd-6fc0-2126ec0b39bd%40postgrespro.ru Back-patch the fix for the InvokeObjectPostAlterHook() call to 9.3 where it was introduced, and the fix for the ObjectAddressSet() call and setting commandCollected to true to 9.5 where those changes to ProcessUtilitySlow() were introduced. http://git.postgresql.org/pg/commitdiff/12bd7dd317e8f4346fb3507578aca790ede6ebea
  • Fix tab completion in psql for ALTER DEFAULT PRIVILEGES. When providing tab completion for ALTER DEFAULT PRIVILEGES, we are including the list of roles as possible options for completion after the GRANT or REVOKE. Further, we accept FOR ROLE/IN SCHEMA at the same time and in either order, but the tab completion was only working for one or the other. Lastly, we weren't using the actual list of allowed kinds of objects for default privileges for completion after the 'GRANT X ON' but instead were completeing to what 'GRANT X ON' supports, which isn't the ssame at all. Address these issues by improving the forward tab-completion for ALTER DEFAULT PRIVILEGES and then constrain and correct how the tail completion is done when it is for ALTER DEFAULT PRIVILEGES. Back-patch the forward/tail tab-completion to 9.6, where we made it easy to handle such cases. For 9.5 and earlier, correct the initial tab-completion to at least be correct as far as it goes and then add a check for GRANT/REVOKE to only tab-complete when the GRANT/REVOKE is the start of the command, so we don't try to do tab-completion after we get to the GRANT/REVOKE part of the ALTER DEFAULT PRIVILEGES command, which is better than providing incorrect completions. Initial patch for master and 9.6 by Gilles Darold, though I cleaned it up and added a few comments. All bugs in the 9.5 and earlier patch are mine. Discussion: https://www.postgresql.org/message-id/1614593c-e356-5b27-6dba-66320a9bc68b@dalibo.com http://git.postgresql.org/pg/commitdiff/f3fd531a51df2a73d8517a542e6999e0186e586b
  • pg_dumpall: Include --verbose option in --help output. The -v/--verbose option was not included in the output from --help for pg_dumpall even though it's in the pg_dumpall documentation and has apparently been around since pg_dumpall was reimplemented in C in 2002. Fix that by adding it. Pointed out by Daniel Westermann. Back-patch to all supported branches. Discussion: https://www.postgresql.org/message-id/2020970042.4589542.1482482101585.JavaMail.zimbra%40dbi-services.com http://git.postgresql.org/pg/commitdiff/86d216c77549e200b95bed487b6fb87d99a1e789

Joe Conway pushed:

  • Improve dblink error message when remote does not provide it. When dblink or postgres_fdw detects an error on the remote side of the connection, it will try to construct a local error message as best it can using libpq's PQresultErrorField(). When no primary message is available, it was bailing out with an unhelpful "unknown error". Make that message better and more style guide compliant. Per discussion on hackers. Backpatch to 9.2 except postgres_fdw which didn't exist before 9.3. Discussion: https://postgr.es/m/19872.1482338965%40sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/ea0aa9698cfa74bb04cf53d813924fe67f278c30
  • Protect dblink from invalid options when using postgres_fdw server. When dblink uses a postgres_fdw server name for its connection, it is possible for the connection to have options that are invalid with dblink (e.g. "updatable"). The recommended way to avoid this problem is to use dblink_fdw servers instead. However there are use cases for using postgres_fdw, and possibly other FDWs, for dblink connection options, therefore protect against trying to use any options that do not apply by using is_valid_dblink_option() when building the connection string from the options. Back-patch to 9.3. Although 9.2 supports FDWs for connection info, is_valid_dblink_option() did not yet exist, and neither did postgres_fdw, at least in the postgres source tree. Given the lack of previous complaints, fixing that seems too invasive/not worth it. Author: Corey Huinker Reviewed-By: Joe Conway Discussion: https://postgr.es/m/CADkLM%3DfWyXVEyYcqbcRnxcHutkP45UHU9WD7XpdZaMfe7S%3DRwA%40mail.gmail.com http://git.postgresql.org/pg/commitdiff/c4448683893bd37b59003603bc9075d362e81b5a
  • Make dblink try harder to form useful error messages. When libpq encounters a connection-level error, e.g. runs out of memory while forming a result, there will be no error associated with PGresult, but a message will be placed into PGconn's error buffer. postgres_fdw takes care to use the PGconn error message when PGresult does not have one, but dblink has been negligent in that regard. Modify dblink to mirror what postgres_fdw has been doing. Back-patch to all supported branches. Author: Joe Conway Reviewed-By: Tom Lane Discussion: https://postgr.es/m/02fa2d90-2efd-00bc-fefc-c23c00eb671e%40joeconway.com http://git.postgresql.org/pg/commitdiff/2f802d95b4904dbed3dfdca1b3a607cd085d2e20
  • Improve RLS documentation with respect to COPY. Documentation for pg_restore said COPY TO does not support row security when in fact it should say COPY FROM. Fix that. While at it, make it clear that "COPY FROM" does not allow RLS to be enabled and INSERT should be used instead. Also that SELECT policies will apply to COPY TO statements. Back-patch to 9.5 where RLS first appeared. Author: Joe Conway Reviewed-By: Dean Rasheed and Robert Haas Discussion: https://postgr.es/m/5744FA24.3030008%40joeconway.com http://git.postgresql.org/pg/commitdiff/0a85c102254b72ec7ce16bc504206a1a5c84bd76

Michael Meskes pushed:

Andres Freund pushed:

Correctifs en attente

Thomas Munro sent in a patch to add support for delta relations in AFTER triggers to PL/PythonU.

Petr Jelínek sent in four more revisions of a patch to add logical replication.

Petr Jelínek sent in a patch to enable logical replication of data which was there before logical replication was turned on.

Michaël Paquier sent in a patch to change detection of corrupted 2PC files as FATAL and minimize window between history file and end-of-recovery record.

Kyotaro HORIGUCHI sent in a patch to protect syscache from bloating with negative cache entries.

Thomas Munro sent in another revision of a patch to allow measuring replay lag.

Beena Emerson sent in a patch to make wal segment size initdb configurable.

Ashutosh Sharma sent in another revision of a patch to add support for hash index in pageinspect contrib module.

Michaël Paquier sent in two revisions of a patch to add tests for authentication and pg_hba.conf.

Heikki Linnakangas sent in a patch to use the "plain:" prefix for plaintext passwords stored in pg_authid.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Ants Aasma and Craig Ringer traded patches to fix an issue where the replication slot xmin is not reset if HS feedback is turned off while the standby is shut down.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous query execution.

Heikki Linnakangas sent in another revision of a patch to add logical tape pause/resume.

Ashutosh Sharma sent in a patch to create a pgstathashindex() function to get hash index table statistics.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Andrea Urbani sent in a patch to add --custom-fetch-table and --custom-fetch-value parameters to pg_dump.

Etsuro Fujita and Ashutosh Bapat traded patches to fix a FDW bug.

Artur Zakirov sent in another revision of a patch to output pg_ts_config_map entries with pg_event_trigger_ddl_commands().

Michaël Paquier sent in a patch to fix a potential data loss of 2PC files.

Ashutosh Bapat sent in a patch to translate partition hierarchy into inheritance hierarchy.

Craig Ringer sent in three revisions of a patch to introduce txid_status(bigint) to get status of an xact.

Pavel Stěhule sent in another revision of a patch to add xmltable().

Kuntal Ghosh sent in a patch to reduce WALWriteLock contention.

Amul Sul sent in another revision of a patch to add a pg_background contrib extension.

Amit Khandekar sent in a patch to implement parallel append.

Claudio Freire sent in a patch to prefetch buffers on backward scan and allow using more than 1GB work mem in VACUUM.

Ãlvaro Herrera sent in two more revisions of a patch to implement indirect indexes.

Craig Ringer sent in a patch to add a new CREATE_REPLICATION_SLOT option.

Daniel Vérité sent in another revision of a patch to improve some of the psql variable hooks.

Jerry Yu sent in a patch to push down quals into EXCEPT.

Fabien COELHO sent in two more revisions of a patch to fix the handling of compound/combined queries by pg_stat_statements.

Joel Jacobson sent in a patch to implement seconds resolution for wait_start.

Rafia Sabih sent in another revision of a patch to implement parallel index-only scans.

Erik Rijkers sent in a patch to fix a typo in a comment in tablecmds.c.

par chl le samedi 7 janvier 2017 à 13h17

mardi 20 décembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 18 décembre 2016

[ndt: Meetup du PLUG à Lyon ce mardi 20 décembre : https://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

PGCon 2017 aura lieu à Ottawa du 23 au 26 mai. Les propositions seront attendues jusqu'au 19 janvier 2017 : http://www.pgcon.org/2017/papers.php

PgConf.Russia 2017 se déroulera du 15 au 17 mars 2017 à Moscou : https://pgconf.ru/en

Le PGDay Asia 2017 s'étendra du 17 au 18 mars à Singapour. L'appel à conférenciers s'éteindra le 16 janvier 2017 : http://tinyurl.com/pgDay-Asia-2017-Cfp

Les inscriptions pour le PGDay du FOSDEM 2017 à Bruxelles sont à présent ouvertes : https://2017.fosdempgday.org/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161219002906.GA8714@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Heikki Linnakangas pushed:

Robert Haas pushed:

  • psql: Fix incorrect version check for table partitioning. Table partitioning was added in 10, not 9.6. Fabrízio de Royes Mello, per report from Jeff Janes http://git.postgresql.org/pg/commitdiff/06e184876bc07c2b1d7144957dcf02c5b4f709c2
  • doc: Fix purported type of pg_am.amhandler to match reality. Joel Jacobson http://git.postgresql.org/pg/commitdiff/b4630e01fd4c73c195025b7307ebc13d489b9ef9
  • Remove should_free arguments to tuplesort routines. Since commit e94568ecc10f2638e542ae34f2990b821bbf90ac, the answer is always "false", and we do not need to complicate the API by arranging to return a constant value. Peter Geoghegan Discussion: http://postgr.es/m/CAM3SWZQWZZ_N=DmmL7tKy_OUjGH_5mN=N=A6h7kHyyDvEhg2DA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/3856cf9607f41245ec9462519c53f1109e781fc5
  • doc: Improve documentation related to table partitioning feature. Commit f0e44751d7175fa3394da2c8f85e3ceb3cdbfe63 implemented table partitioning, but failed to mention the "no row movement" restriction in the documentation. Fix that and a few other issues. Amit Langote, with some additional wordsmithing by me. http://git.postgresql.org/pg/commitdiff/a1a4459c299a86f909c27e391a10d7b9b05ea697
  • Update typedefs.list. So developers can more easily run pgindent locally http://git.postgresql.org/pg/commitdiff/acddbe221b084956a0efd6e4b6c6586e8fd994d7
  • Clean up code, comments, and formatting for table partitioning. Amit Langote, plus pgindent-ing by me. Inspired in part by review comments from Tomas Vondra. http://git.postgresql.org/pg/commitdiff/4b9a98e154cec81849af24091443747a6057c968
  • Fix bug in hashbulkdelete. Commit 6d46f4783efe457f74816a75173eb23ed8930020 failed to account for the possibility that hashbulkdelete() might encounter a bucket that has been split since it began scanning the bucket array. Repair. Extracted from a larger pathc by Amit Kapila; I rewrote the comment. http://git.postgresql.org/pg/commitdiff/501c7b94bcb00cfa0faad60135cf6af82fd56a3a
  • Fix bugs in RelationGetPartitionDispatchInfo. The previous coding was not quite right for cases involving multiple levels of partitioning. Amit Langote http://git.postgresql.org/pg/commitdiff/a25665088d812d08bb888e961f208eaebf522050
  • Remove _hash_wrtbuf() in favor of calling MarkBufferDirty(). The whole concept of _hash_wrtbuf() is that we need to know at the time we're releasing the buffer lock (and pin) whether we dirtied the buffer, but this is easy to get wrong. This patch actually fixes one non-obvious bug of that form: hashbucketcleanup forgot to signal _hash_squeezebucket, which gets the primary bucket page already locked, as to whether it had already dirtied the page. Calling MarkBufferDirty() at the places where we dirty the buffer is more intuitive and lets us simplify the code in various places as well. On top of all that, the ultimate goal here is to make hash indexes WAL-logged, and as the comments to _hash_wrtbuf() note, it should go away when that happens. Making it go away a little earlier than that seems like a good preparatory step. Report by Jeff Janes. Diagnosis by Amit Kapila, Kuntal Ghosh, and Dilip Kumar. Patch by me, after studying an alternative patch submitted by Amit Kapila. Discussion: http://postgr.es/m/CAA4eK1Kf6tOY0oVz_SEdngiNFkeXrA3xUSDPPORQvsWVPdKqnA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/25216c98938495fd741bf585dcbef45b3a9ffd40
  • Fix more hash index bugs around marking buffers dirty. In _hash_freeovflpage(), if we're freeing the overflow page that immediate follows the page to which tuples are being moved (the confusingly-named "write buffer"), don't forget to mark that page dirty after updating its hasho_nextblkno. In _hash_squeezebucket(), it's not necessary to mark the primary bucket page dirty if there are no overflow pages, because there's nothing to squeeze in that case. Amit Kapila, with help from Kuntal Ghosh and Dilip Kumar, after an initial trouble report by Jeff Janes. http://git.postgresql.org/pg/commitdiff/6a4fe1127c5a0ea1515589e416aa29e088170c0e
  • Unbreak Finalize HashAggregate over Partial HashAggregate. Commit 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5 introduced the use of a new type of hash table with linear reprobing for hash aggregates. Such a hash table behaves very poorly if keys are inserted in hash order, which does in fact happen in the case where a query use a Finalize HashAggregate node fed (via Gather) by a Partial HashAggregate node. In fact, queries with this type of plan tend to run effectively forever. Fix that by seeding the hash value differently in each worker (and in the leader, if it participates). Andres Freund and Robert Haas http://git.postgresql.org/pg/commitdiff/b81b5a96f424531b97cdd1dba97d9d1b9c9d372e
  • Simplify LWLock tranche machinery by removing array_base/array_stride. array_base and array_stride were added so that we could identify the offset of an LWLock within a tranche, but this facility is only very marginally used apart from the main tranche. So, give every lock in the main tranche its own tranche ID and get rid of array_base, array_stride, and all that's attached. For debugging facilities (Trace_lwlocks and LWLOCK_STATS) print the pointer address of the LWLock using %p instead of the offset. This is arguably more useful, and certainly a lot cheaper. Drop the offset-within-tranche from the information reported to dtrace and from one can't-happen message inside lwlock.c. The main user-visible impact of this change is that pg_stat_activity will now report all waits for LWLocks as "LWLock" rather than reporting some as "LWLockTranche" and others as "LWLockNamed". The main motivation for this change is that the need to specify an array_base and an array_stride is awkward for parallel query. There is only a very limited supply of tranche IDs so we can't just keep allocating new ones, and if we try to use the same tranche IDs every time then we run into trouble when multiple parallel contexts are use simultaneously. So if we didn't get rid of this mechanism we'd have to make it even more complicated. By simplifying it in this way, we instead reduce the size of the generated code for lwlock.c by about 5%. Discussion: http://postgr.es/m/CA+TgmoYsFn6NUW1x0AZtupJGUAs1UDY4dJtCN47_Q6D0sP80PA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/3761fe3c20bb040b15f0e8da58d824631da00caa
  • Fix outdated comment in lwlock.c Commit 3761fe3c20bb040b15f0e8da58d824631da00caa should have made this change, but didn't. Reported by Ãlvaro Herrera. http://git.postgresql.org/pg/commitdiff/591ccb66d24258f6d1084343b3c33c96e3e2b36d

Tom Lane pushed:

  • Make the different Unix-y semaphore implementations ABI-compatible. Previously, the "sem" field of PGPROC varied in size depending on which kernel semaphore API we were using. That was okay as long as there was only one likely choice per platform, but in the wake of commit ecb0d20a9, that assumption seems rather shaky. It doesn't seem out of the question anymore that an extension compiled against one API choice might be loaded into a postmaster built with another choice. Moreover, this prevents any possibility of selecting the semaphore API at postmaster startup, which might be something we want to do in future. Hence, change PGPROC.sem to be PGSemaphore (i.e. a pointer) for all Unix semaphore APIs, and turn the pointed-to data into an opaque struct whose contents are only known within the responsible modules. For the SysV and unnamed-POSIX APIs, the pointed-to data has to be allocated elsewhere in shared memory, which takes a little bit of rejiggering of the InitShmemAllocation code sequence. (I invented a ShmemAllocUnlocked() function to make that a little cleaner than it used to be. That function is not meant for any uses other than the ones it has now, but it beats having InitShmemAllocation() know explicitly about allocation of space for semaphores and spinlocks.) This change means an extra indirection to access the semaphore data, but since we only touch that when blocking or awakening a process, there shouldn't be any meaningful performance penalty. Moreover, at least for the unnamed-POSIX case on Linux, the sem_t type is quite a bit wider than a pointer, so this reduces sizeof(PGPROC) which seems like a good thing. For the named-POSIX API, there's effectively no change: the PGPROC.sem field was and still is a pointer to something returned by sem_open() in the postmaster's memory space. Document and check the pre-existing limitation that this case can't work in EXEC_BACKEND mode. It did not seem worth unifying the Windows semaphore ABI with the Unix cases, since there's no likelihood of needing ABI compatibility much less runtime switching across those cases. However, we can simplify the Windows code a bit if we define PGSemaphore as being directly a HANDLE, rather than pointer to HANDLE, so let's do that while we're here. (This also ends up being no change in what's physically stored in PGPROC.sem. We're just moving the HANDLE fetch from callees to callers.) It would take a bunch of additional code shuffling to get to the point of actually choosing a semaphore API at postmaster start, but the effects of that would now be localized in the port/XXX_sema.c files, so it seems like fit material for a separate patch. The need for it is unproven as yet, anyhow, whereas the ABI risk to extensions seems real enough. Discussion: https://postgr.es/m/4029.1481413370@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/be7b2848c6d8bdbfb63ab403c535713708c4af52
  • Fix race condition in test_decoding "slot" test. This test, just added in commit a924c327e, sometimes fails because the old backend hasn't finished dropping the temporary replication slot when the new backend looks. Borrow the previously-invented methodology for waiting for the old process to disappear from pg_stat_activity. Petr Jelinek Discussion: https://postgr.es/m/62935e6f-4f1b-c433-e0fa-7f936a38b3e5@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/23f722ba8e19ca1a7c2ada9d6e687989b6e8f4d1
  • Catversion bump for temporary replication slots. Missed in commit a924c327e2793d2025b19e18de7917110dc8afd8. Per Fujii Masao. http://git.postgresql.org/pg/commitdiff/9b3d02c2a9eb93cc4754857361abee449a3fe0cb
  • Fix creative, but unportable, spelling of "ptr != NULL". Or at least I suppose that's what was really meant here. But even aside from the not-per-project-style use of "0" to mean "NULL", I doubt it's safe to assume that all valid pointers are > NULL. Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/563d575fd73361f6118c13f2816988eba8e1f657
  • Prevent planagg.c from failing on queries containing CTEs. The existing tests in preprocess_minmax_aggregates() usually prevent it from trying to do anything with queries containing CTEs, but there's an exception: a CTE could be present as a member of an appendrel, if we flattened a UNION ALL that contains CTE references. If it did try to generate an optimized path for a query using a CTE, it failed with "could not find plan for CTE", as reported by Torsten Förtsch. The proximate cause is an unwise decision in commit 3fc6e2d7f to clear subroot->cte_plan_ids in build_minmax_path(). That left the subroot's cte_plan_ids list out of step with its parse->cteList. Removing the "subroot->cte_plan_ids = NIL;" assignment is enough to let the case work again, but really it's pretty silly to be expending any cycles at all in this module when there are CTEs: we always treat their outputs as unordered so there's no way for the optimization to win. Hence, also add an early-exit test so we don't waste time like that. Back-patch to 9.6 where the misbehavior was introduced. Report: https://postgr.es/m/CAKkG4_=gjY5QiHtqSZyWMwDuTd_CftKoTaCqxjJ7uUz1-Gw=qw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/1f542a2eacca030c676cbb594f3b362d43f2f857
  • Improve handling of array elements as getdiag_targets and cursor_variables. There's no good reason why plpgsql's GET DIAGNOSTICS statement can't support an array element as target variable, since the execution code already uses the generic exec_assign_value() function to assign to it. Hence, refactor the grammar to allow that, by making getdiag_target depend on the assign_var production. Ideally we'd also let a cursor_variable expand to an element of a refcursor[] array, but that's substantially harder since those statements also have to handle bound-cursor-variable cases. For now, just make sure the reported error is sensible, ie "cursor variable must be a simple variable" not "variable must be of type cursor or refcursor". The latter was quite confusing from the user's viewpoint, since what he wrote satisfies the claimed restriction. Per bug #14463 from Zhou Digoal. Given the lack of previous complaints, I see no need for a back-patch. Discussion: https://postgr.es/m/20161213152548.14897.81245@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/55caaaeba877eac1feb6481fb413fa04ae9046ac
  • Sync our copy of the timezone library with IANA release tzcode2016j. This is a trivial update (consisting in fact only in the addition of a comment). The point is just to get back to being synced with an official release of tzcode, rather than some ad-hoc point in their commit history, which is where commit 1f87181e1 left it. http://git.postgresql.org/pg/commitdiff/93513d1b6559b2d0805f0b02d312ee550e3d010b
  • Improve documentation around TS_execute(). I got frustrated by the lack of commentary in this area, so here is some reverse-engineered documentation, along with minor stylistic cleanup. No code changes more significant than removal of unused variables. Back-patch to 9.6, not because that's useful in itself, but because we have some bugs to fix in phrase search and this would cause merge failures if it's only in HEAD. http://git.postgresql.org/pg/commitdiff/23c75b55aaccddea79545ffaf1cbfc9f1edeaa8c
  • Fix FK-based join selectivity estimation for semi/antijoins. This case wasn't thought through sufficiently in commit 100340e2d. It's true that the FK proves that every outer row has a match in the inner table, but we forgot that some of the inner rows might be filtered away by WHERE conditions located within the semijoin's RHS. If the RHS is just one table, we can reasonably take the semijoin selectivity as equal to the fraction of the referenced table's rows that are expected to survive its restriction clauses. If the RHS is a join, it's not clear how much of the referenced table might get through the join, so fall back to the same rule we were already using for other outer-join cases: use the minimum of the regular per-clause selectivity estimates. This gives the same result as if we hadn't considered the FK at all when there's a single FK column, but it should still help for multi-column FKs, which is the case that 100340e2d is really meant to help with. Back-patch to 9.6 where the previous commit came in. Discussion: https://postgr.es/m/16149.1481835103@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/7fa93eec4e0c9c3e801e3c51aa4bae3a38aaa218
  • In contrib/uuid-ossp, #include headers needed for ntohl() and ntohs(). Oversight in commit b8cc8f947. I just noticed this causes compiler warnings on FreeBSD, and it really ought to cause warnings elsewhere too: all references I can find say that <arpa/inet.h> is required for these. We have a lot of code elsewhere that thinks that both <netinet/in.h> and <arpa/inet.h> should be included for these functions, so do it that way here too, even though <arpa/inet.h> ought to be sufficient according to the references I consulted. Back-patch to 9.4 where the previous commit landed. http://git.postgresql.org/pg/commitdiff/4a0a34b5b678f0292d3a85a85fb10c79c393be26

Peter Eisentraut pushed:

Magnus Hagander pushed:

Fujii Masao pushed:

  • Add missing documentation for effective_io_concurrency tablespace option. The description of effective_io_concurrency option was missing in ALTER TABLESPACE docs though it's included in CREATE TABLESPACE one. Back-patch to 9.6 where effective_io_concurrency tablespace option was added. Michael Paquier, reported by Marc-Olaf Jaschke http://git.postgresql.org/pg/commitdiff/4e344c2cf4ff00ca38ea0035bc137dab95fdd0c0
  • Ensure that num_sync is greater than zero in synchronous_standby_names. Previously num_sync could be set to zero and this setting caused an assertion failure. This means that multiple synchronous standbys code should assume that num_sync is greater than zero. Also setting num_sync to zero is nonsense because it's basically the configuration for synchronous replication. If users want not to make transaction commits wait for any standbys, synchronous_standby_names should be emptied to disable synchronous replication instead of setting num_sync to zero. This patch forbids users from setting num_sync to zero in synchronous_standby_names. If zero is specified, an error will happen during processing the parameter settings. Back-patch to 9.6 where multiple synchronous standbys feature was added. Patch by me. Reviewed by Tom Lane. Discussion: <CAHGQGwHWB3izc6cXuFLh5kOcAbFXaRhhgwd-X5PeN9TEjxqXwg@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/93eb619cd35b8adcfe6c86e34ea45d2e8edd322b

Noah Misch pushed:

Correctifs en attente

Pavel Stěhule sent in two more revisions of a patch to implement \g[b]store commands for psql.

Rafia Sabih sent in a patch to implement parallel index-only scans.

Michaël Paquier sent in a patch to fix an issue with strong random on Win32.

Michaël Paquier, Amit Kapila and Ashutosh Sharma traded patches to fix an issue that manifested as a hang in pldebugger.

Amit Kapila sent in another revision of a patch to fix some hash index issues.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Nikita Glukhov sent in a patch to implement recursive processing of nested objects and arrays in json[b]_populate_record[set](), json[b]_to_record[set]().

Tomas Vondra sent in two more revisions of a patch to implement two slab-like memory allocators.

Kyotaro HORIGUCHI sent in another revision of a patch to implement character conversion with radix trees.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Tom Lane sent in a patch to fix matching of boolean index columns to sort ordering.

Aleksander Alekseev sent in a patch to fix documentation of the timestamp type.

Gilles Darold sent in another revision of a patch to implement pg_current_logfile().

Andrew Borodin sent in a patch to add a sleep sort test for pg_background.

Masahiko Sawada and Fujii Masao traded patches to add quorum commit.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Robert Haas sent in a patch to refactor tstate serialization in preparation for making CURRENT_* parallel safe.

Jesper Pedersen sent in a patch to create a pg_catversion() builtin function.

Michaël Paquier, Fujii Masao, and Kyotaro HORIGUCHI traded patches to fix a pg_basebackup crash.

Vladimir Rusinov sent in a patch to rename pg_switch_xlog to pg_switch_wal.

Robert Haas sent in a patch to remove _hash_wrtbuf() and cause those functions which previously called it to do MarkBufferDirty() directly.

Peter Eisentraut, Petr Jelínek, and Steve Singer traded patches to implement logical replication.

Rahila Syed sent in another revision of a patch to turn the types columns with unknown type in views to text.

Dagfinn Ilmari Mannsåker sent in a patch to add GUCs for predicate lock promotion thresholds.

Magnus Hagander sent in a patch to remove the -X option from pg_basebackup.

Magnus Hagander sent in a patch to add missing newlines to error messages in pg_basebackup.

Wesley Massuda sent in a patch to implement composite type NOT NULL constraints.

Alexander Law sent in a patch to speed up documentation builds.

Pavel Stěhule sent in two more revisions of a patch to implement server-side session variables.

Dmitry Ivanov sent in a patch to fix an infelicity between the ancient sql_inheritance GUC and the new native partitions.

Heikki Linnakangas sent in a patch to shorten the window between writing the timeline history file and writing the end-of-recovery record.

Magnus Hagander sent in two revisions of a patch to add a --no-slot option to pg_basebackup.

Heikki Linnakangas sent in a patch to use the "plain:" prefix for plaintext passwords stored in pg_authid.

Amit Kapila sent in two revisions of a patch to fix dirty marking and lock chaining for hash indexes.

Mithun Cy sent in another revision of a patch to cache meta page for hash indexes.

Antonin Houska sent in a patch to simplify some code in basebackup.

Peter Moser sent in another revision of a patch to implement temporal query processing with range types.

Fujii Masao sent in a patch to fix a bug in synchronous replication where there were an invalid number of sync standbys in synchronous_standby_names.

Stephen Frost sent in a patch to fix some infelicities between pg_dump and TRANSFORMs, in passing fixing pg_dump for some ancient versions of PostgreSQL.

Robert Haas sent in a patch to remove the sql_inheritance GUC.

Michaël Paquier sent in a patch to fix some typos in waldender.c and slot.c.

Dmitry Dolgov sent in a patch to implement jsonb_delete with arrays.

Dean Rasheed sent in a patch to fix a CREATE OR REPLACE VIEW bug.

Stas Kelvich sent in a patch to speed up two-phase transactions via a twophase recovery list.

Kevin Grittner sent in another revision of a patch to add transition tables for, among other things, materialized view maintenance and row access in per-statement triggers.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Peter Eisentraut sent in a patch to add a function to import operation system collations.

par chl le mardi 20 décembre 2016 à 02h10

mardi 13 décembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 11 décembre 2016

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • Le PGDay.IT 2016 aura lieu à Prato le 13 décembre 2016 : http://pgday.it
  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161211224221.GB8506@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Fujii Masao pushed:

  • Fix incorrect output from gin_desc(). Previously gin_desc() displayed incorrect output "unknown action 0" for XLOG_GIN_INSERT and XLOG_GIN_VACUUM_DATA_LEAF_PAGE records with valid actions. The cause of this problem was that gin_desc() wrongly used XLogRecGetData() to extract data from those records. Since they were registered by XLogRegisterBufData(), gin_desc() should have used XLogRecGetBlockData(), instead, like gin_redo(). Also there were other differences about how to treat XLOG_GIN_INSERT record between gin_desc() and gin_redo(). This commit fixes gin_desc() routine so that it treats those records in the same way as gin_redo(). Batch-patch to 9.5 where WAL record format was revamped and XLogRegisterBufData() was added. Reported-By: Andres Freund Reviewed-By: Tom Lane Discussion: <20160509194645.7lewnpw647zegx2m@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/5dc851afde8d9ef9947f21799f7a1b08bf0bf812
  • Fix typo in docs. Reported-by: Darko Prelec http://git.postgresql.org/pg/commitdiff/daac8e30eb7874722f277ae3461abe46a39e56ed
  • Improve documentation about pg_stat_replication view. Add the descriptions of possible values in "state" and "sync_state" columns of pg_stat_replication view. Author: Michael Paquier, slightly modified by me Discussion: <CAB7nPqT7APWrvPFZrcjKEHoq4=g3z2ErxtTdojSf+sDALzuemA@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/dfe530a09226a9de80f2b4c3d5f667bf51481c49

Heikki Linnakangas pushed:

  • Fix creation of stand-alone INSTALL.html file. I missed the notice at the top of the file, that plain xref must not be used in installation.sgml. Per buildfarm member guaibasaurus. http://git.postgresql.org/pg/commitdiff/7dd8eb39bd2b9e06eeef038f80ae327efb4a7d55
  • Fix typo in new message in configure. Remove spurious "of", and reformat to fit on a 80 chars wide line. http://git.postgresql.org/pg/commitdiff/44a977f55f33834a2fe0d1d9bd5eeb29ce67e914
  • Fix whitespace. Thomas Munro http://git.postgresql.org/pg/commitdiff/9790b87f594565c11599b2004466169b8c2fd4af
  • Fix query cancellation. In commit fe0a0b59, the datatype used for MyCancelKey and other variables that store cancel keys were changed from long to uint32, but I missed this one. That broke query cancellation on platforms where long is wider than 32 bits. Report by Andres Freund, fix by Michael Paquier. http://git.postgresql.org/pg/commitdiff/81f2e514a9b2d813bb5fd6b62523757aa7a6517f
  • Replace PostmasterRandom() with a stronger source, second attempt. This adds a new routine, pg_strong_random() for generating random bytes, for use in both frontend and backend. At the moment, it's only used in the backend, but the upcoming SCRAM authentication patches need strong random numbers in libpq as well. pg_strong_random() is based on, and replaces, the existing implementation in pgcrypto. It can acquire strong random numbers from a number of sources, depending on what's available: OpenSSL RAND_bytes(), if built with OpenSSL, on Windows, the native cryptographic functions are used, and finally /dev/urandom. Unlike the current pgcrypto function, the source is chosen by configure. That makes it easier to test different implementations, and ensures that we don't accidentally fall back to a less secure implementation, if the primary source fails. All of those methods are quite reliable, it would be pretty surprising for them to fail, so we'd rather find out by failing hard. If no strong random source is available, we fall back to using erand48(), seeded from current timestamp, like PostmasterRandom() was. That isn't cryptographically secure, but allows us to still work on platforms that don't have any of the above stronger sources. Because it's not very secure, the built-in implementation is only used if explicitly requested with --disable-strong-random. This replaces the more complicated Fortuna algorithm we used to have in pgcrypto, which is unfortunate, but all modern platforms have /dev/urandom, so it doesn't seem worth the maintenance effort to keep that. pgcrypto functions that require strong random numbers will be disabled with --disable-strong-random. Original patch by Magnus Hagander, tons of further work by Michael Paquier and me. Discussion: https://www.postgresql.org/message-id/CAB7nPqRy3krN8quR9XujMVVHYtXJ0_60nqgVc6oUk8ygyVkZsA@mail.gmail.com Discussion: https://www.postgresql.org/message-id/CAB7nPqRWkNYRRPJA7-cF+LfroYV10pvjdz6GNvxk-Eee9FypKA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/fe0a0b5993dfe24e4b3bcf52fa64ff41a444b8f1
  • Clean up password authentication code a bit. Commit fe0a0b59, which moved code to do MD5 authentication to a separate CheckMD5Auth() function, left behind a comment that really belongs inside the function, too. Also move the check for db_user_namespace inside the function, seems clearer that way. Now that the md5 salt is passed as argument to md5_crypt_verify, it's a bit silly that it peeks into the Port struct to see if MD5 authentication was used. Seems more straightforward to treat it as an MD5 authentication, if the md5 salt argument is given. And after that, md5_crypt_verify only used the Port argument to look at port->user_name, but that is redundant, because it is also passed as a separate 'role' argument. So remove the Port argument altogether. http://git.postgresql.org/pg/commitdiff/fe7bdf0bf67d8ac360d67fa9740074a2c70e88a4
  • Fix quoting and a compiler warning in dumping partitions. Partition name needs to be quoted in the ATTACH PARTITION command constructed in binary-upgrade mode. Silence compiler warning about set but unused variable, without --enable-cassert. http://git.postgresql.org/pg/commitdiff/2560d244b4c9fc08f1d076b3c40913bec5f7e31f
  • Fix thinko in safeguard for negative availMem. Also, use pass read_buffer_size * numInputTapes rather than just availMem to USEMEM, to be neat. Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/64bc26f90d342ca343f5ba383a97691a58991204
  • Fix accounting of memory needed for merge heap. We allegedly allocated all remaining memory for the read buffers of the sort tapes, but we allocated the merge heap only after that. That means that the allocation of the merge heap was guaranteed to go over the memory limit. Fix by allocating the merge heap first. This makes little difference in practice, because the merge heap is tiny, but let's tidy. While we're at it, add a safeguard for the case that we are already over the limit when allocating the read buffers. That shouldn't happen, but better safe than sorry. The memory accounting error was reported off-list by Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/f7d54f4f7ddf72bf4db1783890b058e758b4b894

Tom Lane pushed:

  • Remove extraneous semicolon from uses of relptr_declare(). If we're going to write a semicolon after calls of relptr_declare(), then we don't need one inside the macro, and removing it suppresses "empty declaration" warnings from pickier compilers (eg pademelon). While at it, we might as well use relptr() inside relptr_declare(), because otherwise that macro would likely go unused altogether. Also improve the comment, which I for one found unclear, and provide a specific example of intended usage. http://git.postgresql.org/pg/commitdiff/3ebf2b45454a5fb74e6516ab4915f7a3d44d544f
  • Put AC_MSG_RESULT() call in the right place. Thinko in ecb0d20a9 --- this needs to go one level further out in the "if" nest. As it stood, nothing got printed in the case of selecting named POSIX semaphores. Cosmetic issue only, but a bug. http://git.postgresql.org/pg/commitdiff/c648f058319a59ad591dd9d1b0c48dfd655d063a
  • Fix unsafe assumption that struct timeval.tv_sec is a "long". It typically is a "long", but it seems possible that on some platforms it wouldn't be. In any case, this silences a compiler warning on OpenBSD (cf buildfarm member curculio). While at it, use snprintf not sprintf. This format string couldn't possibly overrun the supplied buffer, but that doesn't seem like a good reason not to use the safer style. Oversight in commit f828654e1. Back-patch to 9.6 where that came in. http://git.postgresql.org/pg/commitdiff/0645dacc371da6169b06934e3bd238c5f770fe25
  • Handle empty or all-blank PAGER setting more sanely in psql. If the PAGER environment variable is set but contains an empty string, psql would pass it to "sh" which would silently exit, causing whatever query output we were printing to vanish entirely. This is quite mystifying; it took a long time for us to figure out that this was the cause of Joseph Brenner's trouble report. Rather than allowing that to happen, we should treat this as another way to specify "no pager". (We could alternatively treat it as selecting the default pager, but it seems more likely that the former is what the user meant to achieve by setting PAGER this way.) Nonempty, but all-white-space, PAGER values have the same behavior, and it's pretty easy to test for that, so let's handle that case the same way. Most other cases of faulty PAGER values will result in the shell printing some kind of complaint to stderr, which should be enough to diagnose the problem, so we don't need to work harder than this. (Note that there's been an intentional decision not to be very chatty about apparent failure returns from the pager process, since that may happen if, eg, the user quits the pager with control-C or some such. I'd just as soon not start splitting hairs about which exit codes might merit making our own report.) libpq's old PQprint() function was already on board with ignoring empty PAGER values, but for consistency, make it ignore all-white-space values as well. It's been like this a long time, so back-patch to all supported branches. Discussion: https://postgr.es/m/CAFfgvXWLOE2novHzYjmQK8-J6TmHz42G8f3X0SORM44+stUGmw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/18f8f784cbbf96ef77eb8943b466b26605824c14
  • Restore psql's SIGPIPE setting if popen() fails. Ancient oversight in PageOutput(): if popen() fails, we'd better reset the SIGPIPE handler before returning stdout, because ClosePager() won't. Noticed while fixing the empty-PAGER issue. http://git.postgresql.org/pg/commitdiff/b7e1ae2328f7d5a88d8916d78b4561d8ef16f99b
  • Fix reporting of column typmods for multi-row VALUES constructs. expandRTE() and get_rte_attribute_type() reported the exprType() and exprTypmod() values of the expressions in the first row of the VALUES as being the column type/typmod returned by the VALUES RTE. That's fine for the data type, since we coerce all expressions in a column to have the same common type. But we don't coerce them to have a common typmod, so it was possible for rows after the first one to return values that violate the claimed column typmod. This leads to the incorrect result seen in bug #14448 from Hassan Mahmood, as well as some other corner-case misbehaviors. The desired behavior is the same as we use in other type-unification cases: report the common typmod if there is one, but otherwise return -1 indicating no particular constraint. It's cheap for transformValuesClause to determine the common typmod while transforming a multi-row VALUES, but it'd be less cheap for expandRTE() and get_rte_attribute_type() to re-determine that info every time they're asked --- possibly a lot less cheap, if the VALUES has many rows. Therefore, the best fix is to record the common typmods explicitly in a list in the VALUES RTE, as we were already doing for column collations. This looks quite a bit like what we're doing for CTE RTEs, so we can save a little bit of space and code by unifying the representation for those two RTE types. They both now share coltypes/coltypmods/colcollations fields. (At some point it might seem desirable to populate those fields for all RTE types; but right now it looks like constructing them for other RTE types would add more code and cycles than it would save.) The RTE change requires a catversion bump, so this fix is only usable in HEAD. If we fix this at all in the back branches, the patch will need to look quite different. Report: https://postgr.es/m/20161205143037.4377.60754@wrigleys.postgresql.org Discussion: https://postgr.es/m/27429.1480968538@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0b78106cd4651ad7867036a463ec743f6d3d2e86
  • Be more careful about Python refcounts while creating exception objects. PLy_generate_spi_exceptions neglected to do Py_INCREF on the new exception objects, evidently supposing that PyModule_AddObject would do that --- but it doesn't. This left us in a situation where a Python garbage collection cycle could result in deletion of exception object(s), causing server crashes or wrong answers if the exception objects are used later in the session. In addition, PLy_generate_spi_exceptions didn't bother to test for a null result from PyErr_NewException, which at best is inconsistent with the code in PLy_add_exceptions. And PLy_add_exceptions, while it did do Py_INCREF on the exceptions it makes, waited to do that till after some PyModule_AddObject calls, creating a similar risk for failure if garbage collection happened within those calls. To fix, refactor to have just one piece of code that creates an exception object and adds it to the spiexceptions module, bumping the refcount first. Also, let's add an additional refcount to represent the pointer we're going to store in a C global variable or hash table. This should only matter if the user does something weird like delete the spiexceptions Python module, but lack of paranoia has caused us enough problems in PL/Python already. The fact that PyModule_AddObject doesn't do a Py_INCREF of its own explains the need for the Py_INCREF added in commit 4c966d920, so we can improve the comment about that; also, this means we really want to do that before not after the PyModule_AddObject call. The missing Py_INCREF in PLy_generate_spi_exceptions was reported and diagnosed by Rafa de la Torre; the other fixes by me. Back-patch to all supported branches. Discussion: https://postgr.es/m/CA+Fz15kR1OXZv43mDrJb3XY+1MuQYWhx5kx3ea6BRKQp6ezGkg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/9cda81f0056ca488dbd6cded64db1238aed816b2
  • Prevent crash when ts_rewrite() replaces a non-top-level subtree with null. When ts_rewrite()'s replacement argument is an empty tsquery, it's supposed to simplify any operator nodes whose operand(s) become NULL; but it failed to do that reliably, because dropvoidsubtree() only examined the top level of the result tree. Rather than make a second recursive pass, let's just give the responsibility to dofindsubquery() to simplify while it's doing the main replacement pass. Per report from Andreas Seltenreich. Artur Zakirov, with some cosmetic changes by me. Back-patch to all supported branches. Discussion: https://postgr.es/m/8737i01dew.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/0eaaaf00e296c2048b868b7c1d3c12c0eae6dd12

Stephen Frost pushed:

Ãlvaro Herrera pushed:

  • Fix crasher bug in array_position(s) array_position and its cousin array_positions were caching the element type equality function's FmgrInfo without being careful enough to put it in a long-lived context. This is obviously broken but it didn't matter in most cases; only when using arrays of records (involving record_eq) it becomes a problem. The fix is to ensure that the type's equality function's FmgrInfo is cached in the array_position's flinfo->fn_mcxt rather than the current memory context. Apart from record types, the only other case that seems complex enough to possibly cause the same problem are range types. I didn't find a way to reproduce the problem with those, so I only include the test case submitted with the bug report as regression test. Bug report and patch: Junseok Yang Discussion: https://postgr.es/m/CAE+byMupUURYiZ6bKYgMZb9pgV1CYAijJGqWj-90W=nS7uEOeA@mail.gmail.com Backpatch to 9.5, where array_position appeared. http://git.postgresql.org/pg/commitdiff/a73491e5fee88f5db70d69e81fa45060b6ed3682

Correctifs en attente

Michaël Paquier sent in another revision of a patch to enable SIGHUP to reload SSL certificates.

Amit Kapila sent in another revision of a patch to enable write-ahead logging (WAL) for hash indexes.

KaiGai Kohei sent in another revision of a patch to pass down limit bounds for ForeignScan and CustomScan.

Pavel StÄ›hule and Ãlvaro Herrera traded patches to add xmltable().

Michaël Paquier sent in three more revisions of a patch to fix a pgcrypto compilation error caused by stack-allocated EVP_CIPHER_CTX.

Kyotaro HORIGUCHI sent in a patch to change the type for map files not to be void * any more, and in passing clean up convutils.pm.

Masahiko Sawada sent in a patch to reduce the maximum autovacuum_vacuum and analyze_scale_factor from 100 to 1.

Mithun Cy sent in a patch to fix an issue with SSL vs. multi-host URLs in libpq.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Andres Freund sent in a WIP patch to use, among other techniques, JIT compilation, to speed up expression processing and tuple deforming.

Craig Ringer sent in another revision of a patch to enable logical decoding on standby.

Rahila Syed sent in a patch to assign valid collations for SET operations on queries with UNKNOWN types.

Heikki Linnakangas sent in a flock of patches to create the infrastructure for, and implement, SCRAM auth.

Gilles Darold and Karl O. Pinc traded patches to implement pg_current_logfile().

Peter Moser sent in another revision of a patch to implement temporal query processing with range types.

Stephen Frost sent in a patch to fix some infelicities between pg_dump and both TRANSFORMs and CASTs.

Michaël Paquier sent in another revision of a patch to fix how unlogged tables are cleaned up.

Etsuro Fujita sent in another revision of a patch to push more FULL JOINs down when using foreign data wrappers.

Aleksander Alekseev sent in two revisions of a patch to clarify some code by using pg_str_containsonly.

Ashutosh Sharma sent in a patch to fix a bug that cause pldebugger to hang.

Peter Eisentraut and Petr Jelínek traded patches to support logical replication in core.

Masahiko Sawada sent in another revision of a patch to implement quorum commit for multiple synchronous replication.

Amit Langote sent in a patch to document the behavior of UPDATE and DELETE under built-in partitioning.

Fabrízio de Royes Mello sent in a patch to fix a breakage of psql's \d when connecting to older versions of PostgreSQL.

Peter Geoghegan sent in a patch to remove should_free tuplesort routine arguments, avoid copying within tuplesort_gettupleslot(), and add valgrind suppression for logtape_write.

Dilip Kumar sent in a patch to implement parallel merge join.

Amit Kapila sent in a patch to fix some issues in hash indexes.

Tom Lane sent in a patch to back-patch use of unnamed POSIX semaphores for Linux.

Petr Jelínek sent in a patch to skip unnecessary snapshot builds.

Pavel Stěhule sent in a patch to add \gstore \gstore_binary to psql.

Mateusz Stefek sent in a patch to optimize index-only scans with filter conditions.

par chl le mardi 13 décembre 2016 à 00h44

samedi 10 décembre 2016

Guillaume Lelarge

Version 1.3 de mon livre : PostgreSQL - Architecture et notions avancées

Mon livre, PostgreSQL - Architecture et notions avancées, a été écrit pendant le développement de la version 9.5. Il est sorti en version finale un peu après la sortie de la version 9.5, donc en plein développement de la 9.6. À la sortie de la version beta de la 9.6, je me suis mis à travailler sur une mise à jour du livre, incluant des corrections suite à des commentaires qui m'étaient parvenus, ainsi que de nombreux ajouts pour traiter les nouveautés de la version 9.6. Le résultat final est disponible depuis hier. Comme il ne s'agit pas d'une nouvelle édition complète, il est disponible gratuitement en téléchargement aux personnes qui ont acheté une version électronique précédente. Ceux qui commanderaient maintenant la version papier aurait cette nouvelle version, intitulée 1.3.

Évidemment, nous allons continuer la mise à jour du livre pour qu'il ne perde pas en intérêt. Une (vraie) deuxième édition sera disponible en fin d'année prochaine, après la sortie de la version 10. Cette version promet de nombreuses nouveautés très intéressantes, comme tout dernièrement un partitionnement mieux intégré dans le cœur de PostgreSQL.

par Guillaume Lelarge le samedi 10 décembre 2016 à 15h36

vendredi 21 octobre 2016

Nicolas Gollet

PG Session #8 les videos à télécharger

Les vidéos de la "PostgreSQL Session #8" organisée par Dalibo et Oslandia viennent d'être mise en ligne sur youtube !

Afin de vous permettre de visionner ces vidéos partout (dans le train, l'avions, aux fins font de l'Ardèche ;) ), je vous propose de pouvoir les télécharger au format "mp4" directement depuis ce partage one drive.

Au programme 8 conférences :
1. Paul Leroux - Postgres comme NoSql Db - le meilleur des deux mondes
2. Nicolas Ribot - Traitements spatiaux parallélisés pour les gros volumes de données
3. Ludovic Delauné - Point Clouds - modèles, chargement, stockage et manipulation de données
4. Tom van Tilburg - From point clouds and vectordata to 3D models with the help of postgres
5. Giuseppe Broccolo & Julien Rouhaud - BRIN indexes on geospatial database
6. Giulio Calacoci - Step up and face the disaster (pgbarman)
7. Julien Tachoires - Temboard - supervision, administration, dashboard et bien plus
8. Olivier Courtin - De l'analyse spatiale avancée à la GéoDataScience en environnement PostgreSQL - PostGIS

Ces vidéos ont été produites par Dalibo et sont sous licence Creative Commons BY-NC-SA n'hésitez donc pas à les partager!

Merci à Télé Millevaches pour leur travail de captation vidéo et de montage ;)

Bon visionnage à toutes et à tous!

par Nicolas GOLLET le vendredi 21 octobre 2016 à 10h34

jeudi 29 septembre 2016

Sébastien Lardière

PostgreSQL 9.6.0

La version 9.6.0 de PostgreSQL est publiée aujourd'hui. La fonctionnalité la plus notable de cette version majeure est la parallélisation des requêtes. De nombreuses autres fonctionnalités permettent de travailler sur des volumes de données toujours plus important. Quelques informations sur le site de Loxodata :... Lire PostgreSQL 9.6.0

par Sébastien Lardière le jeudi 29 septembre 2016 à 14h16

vendredi 20 mai 2016

Guillaume Lelarge

Quelques nouvelles sur les traductions du manuel

J'ai passé beaucoup de temps ces derniers temps sur la traduction du manuel de PostgreSQL.

  • mise à jour pour les dernières versions mineures
  • corrections permettant de générer les PDF déjà disponibles (9.1, 9.2, 9.3 et 9.4) mais aussi le PDF de la 9.5
  • merge pour la traduction de la 9.6

Elles sont toutes disponibles sur le site docs.postgresql.fr.

De ce fait, la traduction du manuel de la 9.6 peut commencer. Pour les intéressés, c'est par là : https://github.com/gleu/pgdocs_fr/wiki/Translation-9.6

N'hésitez pas à m'envoyer toute question si vous êtes intéressé pour participer.

par Guillaume Lelarge le vendredi 20 mai 2016 à 20h35

dimanche 13 mars 2016

Guillaume Lelarge

Fin de la traduction du manuel de la 9.5

Beaucoup de retard pour cette fois, mais malgré tout, on a fini la traduction du manuel 9.5 de PostgreSQL. Évidemment, tous les manuels ont aussi été mis à jour avec les dernières versions mineures.

N'hésitez pas à me remonter tout problème sur la traduction.

De même, j'ai pratiquement terminé la traduction des applications. Elle devrait être disponible pour la version 9.5.2 (pas de date encore connue).

par Guillaume Lelarge le dimanche 13 mars 2016 à 10h41

mardi 8 mars 2016

Sébastien Lardière

Dates à retenir

Trois dates à retenir autour de PostgreSQL : 17 mars, à Nantes, un premier meetup, dans lequel j'évoquerai les nouveautés de PostgreSQL 9.5. 31 mars, à Paris, où j'essayerai de remonter le fil du temps de bases de données. 31 mai, à Lille, où je plongerai dans les structures du stockage de PostgreSQL. Ces trois dates sont l'occasion... Lire Dates à retenir

par Sébastien Lardière le mardi 8 mars 2016 à 08h30

samedi 6 février 2016

Guillaume Lelarge

Début de la traduction du manuel 9.5

J'ai enfin fini le merge du manuel de la version 9.5. Très peu de temps avant la 9.5, le peu de temps que j'avais étant consacré à mon livre. Mais là, c'est bon, on peut bosser. D'ailleurs, Flavie a déjà commencé et a traduit un paquet de nouveaux fichiers. Mais il reste du boulot. Pour les intéressés, c'est par là : https://github.com/gleu/pgdocs_fr/wiki/Translation-9.5

N'hésitez pas à m'envoyer toute question si vous êtes intéressé pour participer.

par Guillaume Lelarge le samedi 6 février 2016 à 12h18

jeudi 4 février 2016

Rodolphe Quiédeville

Indexer pour rechercher des chaines courtes dans PostgreSQL

Les champs de recherche dans les applications web permettent de se voir propooser des résultats à chaque caractère saisies dans le formulaire, pour éviter de trop solliciter les systèmes de stockage de données, les modules standards permettent de définir une limite basse, la recherche n'étant effective qu'à partir du troisième caractères entrés. Cette limite de 3 caractères s'explique par la possibilité de simplement définir des index trigram dans les bases de données, pour PostgreSQL cela se fait avec l'extension standard pg_trgm, (pour une étude détaillé des trigrams je conseille la lecture de cet article).

Si cette technique a apporté beaucoup de confort dans l'utilisation des formulaires de recherche elle pose néanmoins le problème lorsque que l'on doit rechercher une chaîne de deux caractères, innoportun, contre-productif me direz-vous (je partage assez cet avis) mais imaginons le cas de madame ou monsieur Ba qui sont présent dans la base de données et dont on a oublié de saisir le prénom ou qui n'ont pas de prénom, ils ne pourront jamais remonter dans ces formulaires de recherche, c'est assez fâcheux pour eux.

Nous allons voir dans cet article comment résoudre ce problème, commençons par créer une table avec 50000 lignes de données text aléatoire :

CREATE TABLE blog AS SELECT s, md5(random()::text) as d 
   FROM generate_series(1,50000) s;
~# SELECT * from blog LIMIT 4;
 s |                 d                
---+----------------------------------
 1 | 8fa4044e22df3bb0672b4fe540dec997
 2 | 5be79f21e03e025f00dea9129dc96afa
 3 | 6b1ffca1425326bef7782865ad4a5c5e
 4 | 2bb3d7093dc0fffd5cebacd07581eef0
(4 rows)

Admettons que l'on soit un fan de musique des années 80 et que l'on recherche si il existe dans notre table du texte contenant la chaîne fff.

~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%fff%';

                                             QUERY PLAN                                              
-----------------------------------------------------------------------------------------------------
 Seq Scan on blog  (cost=0.00..1042.00 rows=5 width=37) (actual time=0.473..24.130 rows=328 loops=1)
   Filter: (d ~~ '%fff%'::text)
   Rows Removed by Filter: 49672
 Planning time: 0.197 ms
 Execution time: 24.251 ms
(5 rows)

Sans index on s'en doute cela se traduit pas une lecture séquentielle de la table, ajoutons un index. Pour indexer cette colonne avec un index GIN nous allons utiliser l'opérateur gin_trgm_ops disponible dans l'extension pg_trgm.

~# CREATE EXTENSION pg_trgm;
CREATE EXTENSION
~# CREATE INDEX blog_trgm_idx ON blog USING GIN(d gin_trgm_ops);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%fff%';
                                                       QUERY PLAN                                                        
-------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=16.04..34.46 rows=5 width=37) (actual time=0.321..1.336 rows=328 loops=1)
   Recheck Cond: (d ~~ '%fff%'::text)
   Heap Blocks: exact=222
   ->  Bitmap Index Scan on blog_trgm_idx  (cost=0.00..16.04 rows=5 width=0) (actual time=0.176..0.176 rows=328 loops=1)
         Index Cond: (d ~~ '%fff%'::text)
 Planning time: 0.218 ms
 Execution time: 1.451 ms

Cette fois l'index a pu être utilisé, on note au passage que le temps de requête est réduit d'un facteur 20, mais si l'on souhaite désormais rechercher une chaîne de seulement 2 caractères de nouveau une lecture séquentielle a lieu, notre index trigram devient inefficace pour cette nouvelle recherche.

~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                               QUERY PLAN                                                
---------------------------------------------------------------------------------------------------------
 Seq Scan on blog  (cost=0.00..1042.00 rows=3030 width=37) (actual time=0.016..11.712 rows=5401 loops=1)
   Filter: (d ~~ '%ff%'::text)
   Rows Removed by Filter: 44599
 Planning time: 0.165 ms
 Execution time: 11.968 ms

C'est ici que vont intervenir les index bigram, qui comme leur nom l'index travaille sur des couples et non plus des triplets. En premier nous allons tester pgroonga, packagé pour Debian, Ubuntu, CentOS et d'autres systèmes exotiques vous trouverez toutes les explications pour le mettre en place sur la page d'install du projet.

Les versions packagées de la version 1.0.0 ne supportent actuellement que les versions 9.3 et 9.4, mais les sources viennent d'être taguées 1.0.1 avec le support de la 9.5.

CREATE EXTENSION pgroonga;

La création de l'index se fait ensuite en utilisant

~# CREATE INDEX blog_pgroonga_idx ON blog USING pgroonga(d);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                                           QUERY PLAN                                                            
---------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=27.63..482.51 rows=3030 width=37) (actual time=3.721..5.874 rows=2378 loops=1)
   Recheck Cond: (d ~~ '%ff%'::text)
   Heap Blocks: exact=416
   ->  Bitmap Index Scan on blog_pgroonga_idx  (cost=0.00..26.88 rows=3030 width=0) (actual time=3.604..3.604 rows=2378 loops=1)
         Index Cond: (d ~~ '%ff%'::text)
 Planning time: 0.280 ms
 Execution time: 6.230 ms

On retrouve une utilisation de l'index, avec comme attendu un gain de performance.

Autre solution : pg_bigm qui est dédié plus précisément aux index bigram, l'installation se fait soit à partie de paquets RPM, soit directement depuis les sources avec une explication sur le site, claire et détaillée. pg_bigm supporte toutes les versions depuis la 9.1 jusqu'à 9.5.

~# CREATE INDEX blog_bigm_idx ON blog USING GIN(d gin_bigm_ops);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                                         QUERY PLAN                                                          
-----------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=35.48..490.36 rows=3030 width=37) (actual time=2.121..5.347 rows=5401 loops=1)
   Recheck Cond: (d ~~ '%ff%'::text)
   Heap Blocks: exact=417
   ->  Bitmap Index Scan on blog_bigm_idx  (cost=0.00..34.73 rows=3030 width=0) (actual time=1.975..1.975 rows=5401 loops=1)
         Index Cond: (d ~~ '%ff%'::text)
 Planning time: 4.406 ms
 Execution time: 6.052 ms

Sur une table de 500k tuples la création de l'index prend 6,5 secondes pour bigm contre 4,8 pour pgroonga ; en terme de lecture je n'ai pas trouvé de pattern avec de réelle différence, bien pgroonga s'annonce plus rapide que pg_bigm, ce premier étant plus récent que le second on peut s'attendre à ce qu'il ait profité de l'expérience du second.

Coté liberté les deux projets sont publiés sous licence PostgreSQL license.

La réelle différence entre les deux projets est que Pgroonga est une sous partie du projet global Groonga qui est dédié à la recherche fulltext, il existe par exemple Mgroonga dont vous devinerez aisément la cible, pg_bigm est lui un projet autonome qui n'implémente que les bigram dans PostgreSQL.

Vous avez désormais deux méthodes pour indexer des 2-gram, prenez garde toutefois de ne pas en abuser.

La version 9.4.5 de PostgreSQL a été utilisée pour la rédaction de cet article.

par Rodolphe Quiédeville le jeudi 4 février 2016 à 08h38

lundi 1 février 2016

Guillaume Lelarge

Parution de mon livre : "PostgreSQL, architecture et notions avancées"

Après pratiquement deux ans de travail, mon livre est enfin paru. Pour être franc, c'est assez étonnant de l'avoir entre les mains : un vrai livre, avec une vraie reliure et une vraie couverture, écrit par soi. C'est à la fois beaucoup de fierté et pas mal de questionnements sur la façon dont il va être reçu.

Ceci étant dit, sans savoir si le livre sera un succès en soi, c'est déjà pour moi un succès personnel. Le challenge était de pouvoir écrire un livre de 300 pages sur PostgreSQL, le livre que j'aurais aimé avoir entre les mains quand j'ai commencé à utiliser ce SGBD il y a maintenant plus de 15 ans sous l'impulsion de mon ancien patron.

Le résultat est à la hauteur de mes espérances et les premiers retours sont très positifs. Ce livre apporte beaucoup d'explications sur le fonctionnement et le comportement de PostgreSQL qui, de ce fait, n'est plus cette espèce de boîte noire à exécuter des requêtes. La critique rédigée par Jean-Michel Armand dans le GNU/Linux Magazine France numéro 190 est vraiment très intéressante. Je suis d'accord avec son auteur sur le fait que le début est assez ardu : on plonge directement dans la technique, sans trop montrer comment c'est utilisé derrière, en production. Cette partie-là n'est abordée qu'après. C'est une question que je m'étais posée lors de la rédaction, mais cette question est l'éternel problème de l'oeuf et de la poule ... Il faut commencer par quelque chose : soit on explique la base technique (ce qui est un peu rude), puis on finit par montrer l'application de cette base, soit on fait l'inverse. Il n'y a certainement pas une solution meilleure que l'autre. Le choix que j'avais fait me semble toujours le bon, même maintenant. Mais en effet, on peut avoir deux façons de lire le livre : en commençant par le début ou en allant directement dans les chapitres thématiques.

Je suis déjà prêt à reprendre le travail pour proposer une deuxième édition encore meilleure. Cette nouvelle édition pourrait se baser sur la prochaine version majeure de PostgreSQL, actuellement numérotée 9.6, qui comprend déjà des nouveautés très excitantes. Mais cette édition ne sera réellement intéressante qu'avec la prise en compte du retour des lecteurs de la première édition, pour corriger et améliorer ce qui doit l'être. N'hésitez donc pas à m'envoyer tout commentaire sur le livre, ce sera très apprécié.

par Guillaume Lelarge le lundi 1 février 2016 à 17h57

mercredi 13 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL - 3

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce dernier billet de la série liste quelques paramètres de configuration qui font leur apparition dans cette nouvelle version.Suivi de l'horodatage des COMMITLe paramètre track_commit_timestamp permet de marquer dans les journaux de transactions ("WAL") chaque validation ("COMMIT") avec la date et l'heure du serveur. Ce paramètre est un booléen, et... Lire Version 9.5 de PostgreSQL - 3

par Sébastien Lardière le mercredi 13 janvier 2016 à 15h12

Rodolphe Quiédeville

Index multi colonnes GIN, GIST

Ce billet intéressera tous les utilisateurs de colonnes de type hstore ou json avec PostgreSQL. Bien que celui-ci prenne pour exemple hstore il s'applique également aux colonnes json ou jsonb.

Commençons par créer une table et remplissons là avec 100 000 lignes de données aléatoires. Notre exemple représente des articles qui sont associés à un identifiant de langue (lang_id) et des tags catégorisés (tags), ici chaque article peut être associé à un pays qui sera la Turquie ou l'Islande.

~# CREATE TABLE article (id int4, lang_id int4, tags hstore);
CREATE TABLE
~# INSERT INTO article 
SELECT generate_series(1,10e4::int4), cast(random()*20 as int),
CASE WHEN random() > 0.5 
THEN 'country=>Turquie'::hstore 
WHEN random() > 0.8 THEN 'country=>Islande' ELSE NULL END AS x;
INSERT 0 100000

Pour une recherche efficace des articles dans une langue donnée nous ajountons un index de type B-tree sur la colonne lang_id et un index de type GIN sur la colonne tags.

~# CREATE INDEX ON article(lang_id);
CREATE INDEX
~# CREATE INDEX ON article USING GIN (tags);
CREATE INDEX

Nous avons maintenant nos données et nos index, nous pouvons commencer les recherches. Recherchons tous les articles écrit en français (on considère que l'id du français est le 17), qui sont associés à un pays (ils ont un tag country), et analysons le plan d'exécution.

~# EXPLAIN ANALYZE SELECT * FROM article WHERE lang_id=17 AND tags ? 'country';
                                                                QUERY PLAN                                                                
------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on article  (cost=122.42..141.21 rows=5 width=35) (actual time=12.348..13.912 rows=3018 loops=1)
   Recheck Cond: ((tags ? 'country'::text) AND (lang_id = 17))
   Heap Blocks: exact=663
   ->  BitmapAnd  (cost=122.42..122.42 rows=5 width=0) (actual time=12.168..12.168 rows=0 loops=1)
         ->  Bitmap Index Scan on article_tags_idx  (cost=0.00..12.75 rows=100 width=0) (actual time=11.218..11.218 rows=60051 loops=1)
               Index Cond: (tags ? 'country'::text)
         ->  Bitmap Index Scan on article_lang_id_idx  (cost=0.00..109.42 rows=4950 width=0) (actual time=0.847..0.847 rows=5016 loops=1)
               Index Cond: (lang_id = 17)
 Planning time: 0.150 ms
 Execution time: 14.111 ms
(10 rows)

On a logiquement 2 parcours d'index, suivis d'une opération de combinaison pour obtenir le résultat final. Pour gagner un peu en performance on penserait naturellement à créer un index multi colonnes qui contienne lang_id et tags, mais si vous avez déjà essayé de le faire vous avez eu ce message d'erreur :

~# CREATE INDEX ON article USING GIN (lang_id, tags);
ERROR:  42704: data type integer has no default operator class for access method "gin"
HINT:  You must specify an operator class for the index or define a default operator class for the data type.
LOCATION:  GetIndexOpClass, indexcmds.c:1246

Le HINT donnne une piste intéressante, en effet les index de type GIN ne peuvent pas s'appliquer sur les colonnes de type int4 (et bien d'autres).

La solution réside dans l'utilisation d'une extension standard, qui combine les opérateurs GIN et B-tree, btree-gin, précisons tout de suite qu'il existe l'équivalent btree-gist.

Comme toute extension elle s'installe aussi simplement que :

~# CREATE EXTENSION btree_gin;
CREATE EXTENSION

Désormais nous allons pouvoir créer notre index multi-colonne et rejouer notre requête pour voir la différence.

~# CREATE INDEX ON article USING GIN (lang_id, tags);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM article WHERE lang_id=17 AND tags ? 'country';
                                                             QUERY PLAN                                                              
-------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on article  (cost=24.05..42.84 rows=5 width=35) (actual time=1.983..3.777 rows=3018 loops=1)
   Recheck Cond: ((lang_id = 17) AND (tags ? 'country'::text))
   Heap Blocks: exact=663
   ->  Bitmap Index Scan on article_lang_id_tags_idx  (cost=0.00..24.05 rows=5 width=0) (actual time=1.875..1.875 rows=3018 loops=1)
         Index Cond: ((lang_id = 17) AND (tags ? 'country'::text))
 Planning time: 0.211 ms
 Execution time: 3.968 ms
(7 rows)

A la lecture de ce deuxième explain le gain est explicite, même avec un petit jeu de données le coût estimé est divisé par 3, l'on gagne une lecture d'index et une opération de composition. Maintenant nous pouvons supprimer les 2 autres index pour ne conserver que celui-ci.

par Rodolphe Quiédeville le mercredi 13 janvier 2016 à 14h11

mardi 12 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL - 2

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce deuxième billet évoque donc les nouvelles méthodes internes du moteur, c'est-à-dire de nouveaux outils dont PostgreSQL dispose pour traiter les données.Index BRINIl s'agit d'un nouveau type d'index, créé pour résoudre des problèmes d'accès à de très gros volumes de données ; le cas d'usage est une table de log, dans laquelle les données sont... Lire Version 9.5 de PostgreSQL - 2

par Sébastien Lardière le mardi 12 janvier 2016 à 10h45

vendredi 8 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce premier billet revient donc sur les ajouts au langage SQL de la version 9.5 de PostgreSQL.Ces ajouts portent sur de nombreux champs de fonctionnalités, de la sécurité aux gestions de performances en passant par la gestion des transactions.UPSERTCe mot clé désigne en réalité la possibilité d'intercepter une erreur de clé primaire sur un ordre... Lire Version 9.5 de PostgreSQL

par Sébastien Lardière le vendredi 8 janvier 2016 à 10h49