PostgreSQL La base de donnees la plus sophistiquee au monde.

La planète francophone de PostgreSQL

samedi 6 février 2016

Guillaume Lelarge

Début de la traduction du manuel 9.5

J'ai enfin fini le merge du manuel de la version 9.5. Très peu de temps avant la 9.5, le peu de temps que j'avais étant consacré à mon livre. Mais là, c'est bon, on peut bosser. D'ailleurs, Flavie a déjà commencé et a traduit un paquet de nouveaux fichiers. Mais il reste du boulot. Pour les intéressés, c'est par là : https://github.com/gleu/pgdocs_fr/wiki/Translation-9.5

N'hésitez pas à m'envoyer toute question si vous êtes intéressé pour participer.

par Guillaume Lelarge le samedi 6 février 2016 à 12h18

jeudi 4 février 2016

Rodolphe Quiédeville

Indexer pour rechercher des chaines courtes dans PostgreSQL

Les champs de recherche dans les applications web permettent de se voir propooser des résultats à chaque caractère saisies dans le formulaire, pour éviter de trop solliciter les systèmes de stockage de données, les modules standards permettent de définir une limite basse, la recherche n'étant effective qu'à partir du troisième caractères entrés. Cette limite de 3 caractères s'explique par la possibilité de simplement définir des index trigram dans les bases de données, pour PostgreSQL cela se fait avec l'extension standard pg_trgm, (pour une étude détaillé des trigrams je conseille la lecture de cet article).

Si cette technique a apporté beaucoup de confort dans l'utilisation des formulaires de recherche elle pose néanmoins le problème lorsque que l'on doit rechercher une chaîne de deux caractères, innoportun, contre-productif me direz-vous (je partage assez cet avis) mais imaginons le cas de madame ou monsieur Ba qui sont présent dans la base de données et dont on a oublié de saisir le prénom ou qui n'ont pas de prénom, ils ne pourront jamais remonter dans ces formulaires de recherche, c'est assez fâcheux pour eux.

Nous allons voir dans cet article comment résoudre ce problème, commençons par créer une table avec 50000 lignes de données text aléatoire :

CREATE TABLE blog AS SELECT s, md5(random()::text) as d 
   FROM generate_series(1,50000) s;
~# SELECT * from blog LIMIT 4;
 s |                 d                
---+----------------------------------
 1 | 8fa4044e22df3bb0672b4fe540dec997
 2 | 5be79f21e03e025f00dea9129dc96afa
 3 | 6b1ffca1425326bef7782865ad4a5c5e
 4 | 2bb3d7093dc0fffd5cebacd07581eef0
(4 rows)

Admettons que l'on soit un fan de musique des années 80 et que l'on recherche si il existe dans notre table du texte contenant la chaîne fff.

~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%fff%';

                                             QUERY PLAN                                              
-----------------------------------------------------------------------------------------------------
 Seq Scan on blog  (cost=0.00..1042.00 rows=5 width=37) (actual time=0.473..24.130 rows=328 loops=1)
   Filter: (d ~~ '%fff%'::text)
   Rows Removed by Filter: 49672
 Planning time: 0.197 ms
 Execution time: 24.251 ms
(5 rows)

Sans index on s'en doute cela se traduit pas une lecture séquentielle de la table, ajoutons un index. Pour indexer cette colonne avec un index GIN nous allons utiliser l'opérateur gin_trgm_ops disponible dans l'extension pg_trgm.

~# CREATE EXTENSION pg_trgm;
CREATE EXTENSION
~# CREATE INDEX blog_trgm_idx ON blog USING GIN(d gin_trgm_ops);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%fff%';
                                                       QUERY PLAN                                                        
-------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=16.04..34.46 rows=5 width=37) (actual time=0.321..1.336 rows=328 loops=1)
   Recheck Cond: (d ~~ '%fff%'::text)
   Heap Blocks: exact=222
   ->  Bitmap Index Scan on blog_trgm_idx  (cost=0.00..16.04 rows=5 width=0) (actual time=0.176..0.176 rows=328 loops=1)
         Index Cond: (d ~~ '%fff%'::text)
 Planning time: 0.218 ms
 Execution time: 1.451 ms

Cette fois l'index a pu être utilisé, on note au passage que le temps de requête est réduit d'un facteur 20, mais si l'on souhaite désormais rechercher une chaîne de seulement 2 caractères de nouveau une lecture séquentielle a lieu, notre index trigram devient inefficace pour cette nouvelle recherche.

~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                               QUERY PLAN                                                
---------------------------------------------------------------------------------------------------------
 Seq Scan on blog  (cost=0.00..1042.00 rows=3030 width=37) (actual time=0.016..11.712 rows=5401 loops=1)
   Filter: (d ~~ '%ff%'::text)
   Rows Removed by Filter: 44599
 Planning time: 0.165 ms
 Execution time: 11.968 ms

C'est ici que vont intervenir les index bigram, qui comme leur nom l'index travaille sur des couples et non plus des triplets. En premier nous allons tester pgroonga, packagé pour Debian, Ubuntu, CentOS et d'autres systèmes exotiques vous trouverez toutes les explications pour le mettre en place sur la page d'install du projet.

Les versions packagées de la version 1.0.0 ne supportent actuellement que les versions 9.3 et 9.4, mais les sources viennent d'être taguées 1.0.1 avec le support de la 9.5.

CREATE EXTENSION pgroonga;

La création de l'index se fait ensuite en utilisant

~# CREATE INDEX blog_pgroonga_idx ON blog USING pgroonga(d);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                                           QUERY PLAN                                                            
---------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=27.63..482.51 rows=3030 width=37) (actual time=3.721..5.874 rows=2378 loops=1)
   Recheck Cond: (d ~~ '%ff%'::text)
   Heap Blocks: exact=416
   ->  Bitmap Index Scan on blog_pgroonga_idx  (cost=0.00..26.88 rows=3030 width=0) (actual time=3.604..3.604 rows=2378 loops=1)
         Index Cond: (d ~~ '%ff%'::text)
 Planning time: 0.280 ms
 Execution time: 6.230 ms

On retrouve une utilisation de l'index, avec comme attendu un gain de performance.

Autre solution : pg_bigm qui est dédié plus précisément aux index bigram, l'installation se fait soit à partie de paquets RPM, soit directement depuis les sources avec une explication sur le site, claire et détaillée. pg_bigm supporte toutes les versions depuis la 9.1 jusqu'à 9.5.

~# CREATE INDEX blog_bigm_idx ON blog USING GIN(d gin_bigm_ops);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                                         QUERY PLAN                                                          
-----------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=35.48..490.36 rows=3030 width=37) (actual time=2.121..5.347 rows=5401 loops=1)
   Recheck Cond: (d ~~ '%ff%'::text)
   Heap Blocks: exact=417
   ->  Bitmap Index Scan on blog_bigm_idx  (cost=0.00..34.73 rows=3030 width=0) (actual time=1.975..1.975 rows=5401 loops=1)
         Index Cond: (d ~~ '%ff%'::text)
 Planning time: 4.406 ms
 Execution time: 6.052 ms

Sur une table de 500k tuples la création de l'index prend 6,5 secondes pour bigm contre 4,8 pour pgroonga ; en terme de lecture je n'ai pas trouvé de pattern avec de réelle différence, bien pgroonga s'annonce plus rapide que pg_bigm, ce premier étant plus récent que le second on peut s'attendre à ce qu'il ait profité de l'expérience du second.

Coté liberté les deux projets sont publiés sous licence PostgreSQL license.

La réelle différence entre les deux projets est que Pgroonga est une sous partie du projet global Groonga qui est dédié à la recherche fulltext, il existe par exemple Mgroonga dont vous devinerez aisément la cible, pg_bigm est lui un projet autonome qui n'implémente que les bigram dans PostgreSQL.

Vous avez désormais deux méthodes pour indexer des 2-gram, prenez garde toutefois de ne pas en abuser.

La version 9.4.5 de PostgreSQL a été utilisée pour la rédaction de cet article.

par Rodolphe Quiédeville le jeudi 4 février 2016 à 08h38

mercredi 3 février 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 31 janvier 2016

"5432 ... Meet us!" aura lieu à Milan (Italie) les 28 & 29 juin 2016. L'appel à conférenciers court jusqu'au 28 février : http://5432meet.us/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016 : http://pgday.in
  • La première conférence PostgreSQL pan-asiatique se tiendra les 16 et 17 mars 2016 à Singapour : http://2016.pgday.asia/
  • Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. Les inscriptions sont encore ouvertes : http://2016.nordicpgday.org/
  • La 8ème Session PostgreSQL aura lieu le 6 avril 2016 à Lyon (France). L'appel à conférenciers court jusqu'au 29 février à l'adresse call-for-paper AT postgresql-sessions POINT org.
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York : http://www.pgconf.us/2016/
  • La LinuxFest Northwest aura lieu les 23 et 24 avril 2016 au Collège Technique de Bellingham (Washington, USA). L'appel à conférenciers est maintenant lancé : http://www.linuxfestnorthwest.org/2016/present
  • FOSS4G NA (Free and Open Source Software for Geospatial - North America) se tiendra à Raleigh, en Caroline du Nord, du 2 au 5 mai 2016. L'appel à conférenciers est lancé : https://2016.foss4g-na.org/cfp
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa : http://www.pgcon.org/
  • Le PGDay suisse sera, cette année, tenue à l'Université des Sciences Appliquées (HSR) de Rapperswil le 24 juin 2016. L'appel à conférenciers est lancé : http://www.pgday.ch/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20160201051108.GA654@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tatsuo Ishii pushed:

Kevin Grittner pushed:

Tom Lane pushed:

  • Improve ResourceOwners' behavior for large numbers of owned objects. The original coding was quite fast so long as objects were always released in reverse order of addition; otherwise, it degenerated into O(N^2) behavior due to searching for the array element to delete. Improve matters by switching to hashed storage when the number of objects of a given type exceeds 64. (The cutover point is open to discussion, of course, but some simple performance testing suggests that hashing has enough overhead to be a loser below there.) Also, refactor resowner.c so that we don't need N copies of the array management code. Since all the resource IDs the code currently needs to deal with are either pointers or integers, it seems sufficient to create a one-size-fits-all infrastructure in which everything is converted to a Datum for storage. Aleksander Alekseev, reviewed by Stas Kelvich, further fixes by me http://git.postgresql.org/pg/commitdiff/cc988fbb0bf60a83b628b5615e6bade5ae9ae6f4
  • Fix startup so that log prefix %h works for the log_connections message. We entirely randomly chose to initialize port->remote_host just after printing the log_connections message, when we could perfectly well do it just before, allowing %h and %r to work for that message. Per gripe from Artem Tomyuk. http://git.postgresql.org/pg/commitdiff/b8682a7155bee06667c5773e1ca6499a670338b0
  • Fix incorrect pattern-match processing in psql's \det command. listForeignTables' invocation of processSQLNamePattern did not match up with the other ones that handle potentially-schema-qualified names; it failed to make use of pg_table_is_visible() and also passed the name arguments in the wrong order. Bug seems to have been aboriginal in commit 0d692a0dc9f0e532. It accidentally sort of worked as long as you didn't inquire too closely into the behavior, although the silliness was later exposed by inconsistencies in the test queries added by 59efda3e50ca4de6 (which I probably should have questioned at the time, but didn't). Per bug #13899 from Reece Hart. Patch by Reece Hart and Tom Lane. Back-patch to all affected branches. http://git.postgresql.org/pg/commitdiff/7e22470471e9ed7010fcbc4a18b0a461d088d7c7

Ãlvaro Herrera pushed:

Fujii Masao pushed:

Robert Haas pushed:

  • Fix cross-version pg_dump for aggregate combine functions. Fixes a defect in commit a7de3dc5c346e07e0439275982569996e645b3c2. David Rowley, per report from Jeff Janes, who also checked that the fix works. http://git.postgresql.org/pg/commitdiff/025b2f339260b727e113a01a20b616a336bff00a
  • Assert that create_unique_path returns non-NULL. Per off-list discussion with Tom Lane and Michael Paquier, Coverity gets unhappy if this is not done. http://git.postgresql.org/pg/commitdiff/eaf7b1f6432480e93d8c6824fbd503761a1c1a4f
  • Add [NO]BYPASSRLS options to CREATE USER and ALTER USER docs. Patch-by: Filip RembiaÅ‚kowski Reviewed-by: Robert Haas Backpatch-through: 9.5 http://git.postgresql.org/pg/commitdiff/80db1ca2d79338c35bb3e01f2aecad78c2231b06
  • Avoid multiple foreign server connections when all use same user mapping. Previously, postgres_fdw's connection cache was keyed by user OID and server OID, but this can lead to multiple connections when it's not really necessary. In particular, if all relevant users are mapped to the public user mapping, then their connection options are certainly the same, so one connection can be used for all of them. While we're cleaning things up here, drop the "server" argument to GetConnection(), which isn't really needed. This saves a few cycles because callers no longer have to look this up; the function itself does, but only when establishing a new connection, not when reusing an existing one. Ashutosh Bapat, with a few small changes by me. http://git.postgresql.org/pg/commitdiff/96198d94cb7adc664bda341842dc8db671d8be72
  • Add missing quotation mark. This fix accidentally got left out of the previous commit. http://git.postgresql.org/pg/commitdiff/2f6b041f76e6de0fa2921131a23bda794ffb83bb
  • Only try to push down foreign joins if the user mapping OIDs match. Previously, the foreign join pushdown infrastructure left the question of security entirely up to individual FDWs, but it would be easy for a foreign data wrapper to inadvertently open up subtle security holes that way. So, make it the core code's job to determine which user mapping OID is relevant, and don't attempt join pushdown unless it's the same for all relevant relations. Per a suggestion from Tom Lane. Shigeru Hanada and Ashutosh Bapat, reviewed by Etsuro Fujita and KaiGai Kohei, with some further changes by me. http://git.postgresql.org/pg/commitdiff/fbe5a3fb73102c2cfec11aaaa4a67943f4474383
  • postgres_fdw: Refactor deparsing code for locking clauses. The upcoming patch to allow join pushdown in postgres_fdw needs to use this code multiple times, which requires moving it to deparse.c. That seems like a good idea anyway, so do that now both on general principle and to simplify the future patch. Inspired by a patch by Shigeru Hanada and Ashutosh Bapat, but I did it a little differently than what that patch did. http://git.postgresql.org/pg/commitdiff/b88ef201d46e6519b5e0589358c952a4c0f5bf0f
  • Migrate PGPROC's backendLock into PGPROC itself, using a new tranche. Previously, each PGPROC's backendLock was part of the main tranche, and the PGPROC just contained a pointer. Now, the actual LWLock is part of the PGPROC. As with previous, similar patches, this makes it significantly easier to identify these lwlocks in LWLOCK_STATS or Trace_lwlocks output and improves modularity. Author: Ildus Kurbangaliev Reviewed-by: Amit Kapila, Robert Haas http://git.postgresql.org/pg/commitdiff/b319356f0e94a6482c726cf4af96597c211d8d6e
  • Migrate replication slot I/O locks into a separate tranche. This is following in a long train of similar changes and for the same reasons - see b319356f0e94a6482c726cf4af96597c211d8d6e and fe702a7b3f9f2bc5bf6d173166d7d55226af82c8 inter alia. Author: Amit Kapila Reviewed-by: Alexander Korotkov, Robert Haas http://git.postgresql.org/pg/commitdiff/2251179e6ad3a865d2f55e1832fab34608fcce43
  • postgres_fdw: More preliminary refactoring for upcoming join pushdown. The code that generates a complete SQL query for a given foreign relation was repeated in two places, and they didn't quite agree: the EXPLAIN case left out the locking clause. Centralize the code so we get the same behavior everywhere, and adjust calling conventions and which functions are static vs. extern accordingly . Centralize the code so we get the same behavior everywhere, and adjust calling conventions and which functions are static vs. extern accordingly. Ashutosh Bapat, reviewed and slightly adjusted by me. http://git.postgresql.org/pg/commitdiff/cc592c48c58d9c1920f8e2063756dcbcce79e4dd

Peter Eisentraut pushed:

Andrew Dunstan pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Michael Paquier sent in a patch to ensure that fsyncs actually fsync in the case of renaming.

Corey Huinker and Michael Paquier traded revisions of a patch to add generate_series(date,date) and generate_series(date,date,integer).

Corey Huinker sent in two more revisions of a patch to allow limiting FETCH by bytes.

Kaigai Kouhei sent in another revision of a patch to add CustomScan support and an example.

Vinayak Pokale sent in two more revisions of a patch to implement a vacuum progress checker.

Amit Kapila sent in two more revisions of a patch to expand pg_stat_activity to chow waiting.

Pavel Stěhule sent in two more revisions of a patch to add pg_size_bytes().

Pavel Stěhule sent in another revision of a patch to add num_notnulls().

Stas Kelvich sent in a patch to speed up two-phase commits by reading state files into memory during the replay of prepare, and if checkpoint/restartpoint occurs between prepare and commit, to move the data to files.

Craig Ringer sent in two more revisions of a patch to implement failover slots.

SAWADA Masahiko sent in two revisions of a patch to check for a suitable index to use in REFRESH MATERIALIZED VIEW CONCURRENTLY.

Kyotaro HORIGUCHI sent in another revision of a patch to allow async-capable nodes to run the node before ExecProcNode().

Pavel Stěhule and Iacob Catalin traded patches to add an ereport function to PL/PythonU.

Anastasia Lubennikova sent in another revision of a patch to add covering unique indexes.

Etsuro Fujita sent in a patch to fix some capitalizations in fdwhandlers.sgml.

Erik Rijkers sent in a patch to fix some typos in the pgbench docs.

Aleksander Alekseev sent in another revision of a patch to optimize dynahashes.

Stas Kelvich sent in another revision of a patch to add tsvector editing functions.

Pavel Stěhule sent in another revision of a patch to better support %TYPE in PL/pgsql.

José Arthur Benetasso Villanova sent in another revision of a patch to log operating system usernames connecting via unix socket.

Fabien COELHO sent in a patch to fix a couple of minor bugs in pgbench.

Ashutosh Bapat sent in a patch to keep from making separate DB connections to remote servers when not needed in FDWs.

Artur Zakirov sent in a patch to fix some infelicities in tsearch2 parsing.

Petr Jelínek sent in another revision of a patch to add a sequence access method.

Peter Eisentraut sent in two more revisions of a patch to integrate better with systemd.

Michael Paquier sent in three more revisions of a patch to avoid unneeded checkpoints.

Noah Misch sent in two revisions of a patch to fix an issue that could cause a backend crash with nested CREATE TEMP TABLE invocations.

Fabien COELHO and Michael Paquier traded patches to extend pgbench expressions with functions.

Amit Kapila sent in a patch to reduce the number of WAL writes.

Dilip Kumar sent in another revision of a patch to help scale relation extensions.

Artur Zakirov sent in two more revisions of a patch to improve Hunspell dictionary support.

Michael Paquier sent in a patch to fix some comment typos.

Etsuro Fujita sent in another revision of a patch to make foreign table writes more efficient.

Kaigai Kouhei sent in two more revisions of a patch to add CustomScan under a Gather node.

Anastasia Lubennikova sent in three more revisions of a patch to compress B-Trees in a way that's analogous to that just used for GIN.

Ashutosh Bapat sent in two more revisions of a patch to make read operations on the PostgreSQL FDW more efficient.

Fujii Masao and SAWADA Masahiko traded patches to fix tab completion options for SET/RESET in psql.

Alexander Korotkov sent in another revision of a patch to enable creating access methods with an example for Bloom filters.

Alexander Korotkov sent in a patch to refactor SLRU tranches.

Ashutosh Bapat sent in a patch to move the code to deparse SELECT statements into a function deparseSelectStmtForRel().

Vitaly Burovoy sent in a patch to fix an overflow in EXTRACT.

Vik Fearing sent in another revision of a patch to add an idle-in-transaction timeout.

Thomas Munro sent in a patch to detect SSI conflicts before reporting constraint violations.

Vitaly Burovoy sent in a patch to make the behavior of all versions of the "isinf" function be similar.

par N Bougain le mercredi 3 février 2016 à 21h29

lundi 1 février 2016

Guillaume Lelarge

Parution de mon livre : "PostgreSQL, architecture et notions avancées"

Après pratiquement deux ans de travail, mon livre est enfin paru. Pour être franc, c'est assez étonnant de l'avoir entre les mains : un vrai livre, avec une vraie reliure et une vraie couverture, écrit par soi. C'est à la fois beaucoup de fierté et pas mal de questionnements sur la façon dont il va être reçu.

Ceci étant dit, sans savoir si le livre sera un succès en soi, c'est déjà pour moi un succès personnel. Le challenge était de pouvoir écrire un livre de 300 pages sur PostgreSQL, le livre que j'aurais aimé avoir entre les mains quand j'ai commencé à utiliser ce SGBD il y a maintenant plus de 15 ans sous l'impulsion de mon ancien patron.

Le résultat est à la hauteur de mes espérances et les premiers retours sont très positifs. Ce livre apporte beaucoup d'explications sur le fonctionnement et le comportement de PostgreSQL qui, de ce fait, n'est plus cette espèce de boîte noire à exécuter des requêtes. La critique rédigée par Jean-Michel Armand dans le GNU/Linux Magazine France numéro 190 est vraiment très intéressante. Je suis d'accord avec son auteur sur le fait que le début est assez ardu : on plonge directement dans la technique, sans trop montrer comment c'est utilisé derrière, en production. Cette partie-là n'est abordée qu'après. C'est une question que je m'étais posée lors de la rédaction, mais cette question est l'éternel problème de l'oeuf et de la poule ... Il faut commencer par quelque chose : soit on explique la base technique (ce qui est un peu rude), puis on finit par montrer l'application de cette base, soit on fait l'inverse. Il n'y a certainement pas une solution meilleure que l'autre. Le choix que j'avais fait me semble toujours le bon, même maintenant. Mais en effet, on peut avoir deux façons de lire le livre : en commençant par le début ou en allant directement dans les chapitres thématiques.

Je suis déjà prêt à reprendre le travail pour proposer une deuxième édition encore meilleure. Cette nouvelle édition pourrait se baser sur la prochaine version majeure de PostgreSQL, actuellement numérotée 9.6, qui comprend déjà des nouveautés très excitantes. Mais cette édition ne sera réellement intéressante qu'avec la prise en compte du retour des lecteurs de la première édition, pour corriger et améliorer ce qui doit l'être. N'hésitez donc pas à m'envoyer tout commentaire sur le livre, ce sera très apprécié.

par Guillaume Lelarge le lundi 1 février 2016 à 17h57

jeudi 28 janvier 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 24 janvier 2016

La 8ème Session PostgreSQL aura lieu le 6 avril 2016 à Lyon (France). L'appel à conférenciers court jusqu'au 29 février à l'adresse call-for-paper AT postgresql-sessions POINT org.

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Les inscriptions sont encore ouvertes : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016 : http://pgday.in
  • La première conférence PostgreSQL pan-asiatique se tiendra les 16 et 17 mars 2016 à Singapour : http://2016.pgday.asia/
  • Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. Les inscriptions sont encore ouvertes : http://2016.nordicpgday.org/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • La LinuxFest Northwest aura lieu les 23 et 24 avril 2016 au Collège Technique de Bellingham (Washington, USA). L'appel à conférenciers est maintenant lancé : http://www.linuxfestnorthwest.org/2016/present
  • FOSS4G NA (Free and Open Source Software for Geospatial - North America) se tiendra à Raleigh, en Caroline du Nord, du 2 au 5 mai 2016. L'appel à conférenciers est lancé : https://2016.foss4g-na.org/cfp
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa : http://www.pgcon.org/
  • Le PGDay suisse sera, cette année, tenue à l'Université des Sciences Appliquées (HSR) de Rapperswil le 24 juin 2016. L'appel à conférenciers est lancé : http://www.pgday.ch/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20160125053004.GA10201@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Re-pgindent a few files. In preparation for landing index AM interface changes. http://git.postgresql.org/pg/commitdiff/8d290c8ec6c182a4df1d089c21fe84c7912f01fe
  • Restructure index access method API to hide most of it at the C level. This patch reduces pg_am to just two columns, a name and a handler function. All the data formerly obtained from pg_am is now provided in a C struct returned by the handler function. This is similar to the designs we've adopted for FDWs and tablesample methods. There are multiple advantages. For one, the index AM's support functions are now simple C functions, making them faster to call and much less error-prone, since the C compiler can now check function signatures. For another, this will make it far more practical to define index access methods in installable extensions. A disadvantage is that SQL-level code can no longer see attributes of index AMs; in particular, some of the crosschecks in the opr_sanity regression test are no longer possible from SQL. We've addressed that by adding a facility for the index AM to perform such checks instead. (Much more could be done in that line, but for now we're content if the amvalidate functions more or less replace what opr_sanity used to do.) We might also want to expose some sort of reporting functionality, but this patch doesn't do that. Alexander Korotkov, reviewed by Petr Jelínek, and rather heavily editorialized on by me. http://git.postgresql.org/pg/commitdiff/65c5fcd353a859da9e61bfb2b92a99f12937de3b
  • Add explicit cast to amcostestimate call. My compiler doesn't complain here, but David Rowley's does ... http://git.postgresql.org/pg/commitdiff/49b49506502026a3653bca490c939dc8934afe95
  • Fix assorted inconsistencies in GiST opclass support function declarations. The conventions specified by the GiST SGML documentation were widely ignored. For example, the strategy-number argument for "consistent" and "distance" functions is specified to be a smallint, but most of the built-in support functions declared it as an integer, and for that matter the core code passed it using Int32GetDatum not Int16GetDatum. None of that makes any real difference at runtime, but it's quite confusing for newcomers to the code, and it makes it very hard to write an amvalidate() function that checks support function signatures. So let's try to instill some consistency here. Another similar issue is that the "query" argument is not of a single well-defined type, but could have different types depending on the strategy (corresponding to search operators with different righthand-side argument types). Some of the functions threw up their hands and declared the query argument as being of "internal" type, which surely isn't right ("any" would have been more appropriate); but the majority position seemed to be to declare it as being of the indexed data type, corresponding to a search operator with both input types the same. So I've specified a convention that that's what to do always. Also, the result of the "union" support function actually must be of the index's storage type, but the documentation suggested declaring it to return "internal", and some of the functions followed that. Standardize on telling the truth, instead. Similarly, standardize on declaring the "same" function's inputs as being of the storage type, not "internal". Also, somebody had forgotten to add the "recheck" argument to both the documentation of the "distance" support function and all of their SQL declarations, even though the C code was happily using that argument. Clean that up too. Fix up some other omissions in the docs too, such as documenting that union's second input argument is vestigial. So far as the errors in core function declarations go, we can just fix pg_proc.h and bump catversion. Adjusting the erroneous declarations in contrib modules is more debatable: in principle any change in those scripts should involve an extension version bump, which is a pain. However, since these changes are purely cosmetic and make no functional difference, I think we can get away without doing that. http://git.postgresql.org/pg/commitdiff/9ff60273e35cad6e9d3a4adf59d5c2455afe9d9e
  • Fix assorted inconsistencies in GIN opclass support function declarations. GIN had some minor issues too, mostly using "internal" where something else would be more appropriate. I went with the same approach as in 9ff60273e35cad6e, namely preferring the opclass' indexed datatype for arguments that receive an operator RHS value, even if that's not necessarily what they really are. Again, this is with an eye to having a uniform rule for ginvalidate() to check support function signatures. http://git.postgresql.org/pg/commitdiff/dbe2328959e12701fade6b500ad411271923d6e4
  • Add defenses against putting expanded objects into Const nodes. Putting a reference to an expanded-format value into a Const node would be a bad idea for a couple of reasons. It'd be possible for the supposedly immutable Const to change value, if something modified the referenced variable ... in fact, if the Const's reference were R/W, any function that has the Const as argument might itself change it at runtime. Also, because datumIsEqual() is pretty simplistic, the Const might fail to compare equal to other Consts that it should compare equal to, notably including copies of itself. This could lead to unexpected planner behavior, such as "could not find pathkey item to sort" errors or inferior plans. I have not been able to find any way to get an expanded value into a Const within the existing core code; but Paul Ramsey was able to trigger the problem by writing a datatype input function that returns an expanded value. The best fix seems to be to establish a rule that varlena values being placed into Const nodes should be passed through pg_detoast_datum(). That will do nothing (and cost little) in normal cases, but it will flatten expanded values and thereby avoid the above problems. Also, it will convert short-header or compressed values into canonical format, which will avoid possible unexpected lack-of-equality issues for those cases too. And it provides a last-ditch defense against putting a toasted value into a Const, which we already knew was dangerous, cf commit 2b0c86b66563cf2f. (In the light of this discussion, I'm no longer sure that that commit provided 100% protection against such cases, but this fix should do it.) The test added in commit 65c3d05e18e7c530 to catch datatype input functions with unstable results would fail for functions that returned expanded values; but it seems a bit uncharitable to deem a result unstable just because it's expressed in expanded form, so revise the coding so that we check for bitwise equality only after applying pg_detoast_datum(). That's a sufficient condition anyway given the new rule about detoasting when forming a Const. Back-patch to 9.5 where the expanded-object facility was added. It's possible that this should go back further; but in the absence of clear evidence that there's any live bug in older branches, I'll refrain for now. http://git.postgresql.org/pg/commitdiff/b99551832e79c915e4d877cf0a072120bd248748
  • Suppress compiler warning. Given the limited range of i, these shifts should not cause any problem, but that apparently doesn't stop some compilers from whining about them. David Rowley http://git.postgresql.org/pg/commitdiff/d9b9289c837a98b78b948b597fabd9ab0a96c0db
  • Improve index AMs' opclass validation procedures. The amvalidate functions added in commit 65c5fcd353a859da were on the crude side. Improve them in a few ways: * Perform signature checking for operators and support functions. * Apply more thorough checks for missing operators and functions, where possible. * Instead of reporting problems as ERRORs, report most problems as INFO messages and make the amvalidate function return FALSE. This allows more than one problem to be discovered per run. * Report object names rather than OIDs, and work a bit harder on making the messages understandable. Also, remove a few more opr_sanity regression test queries that are now superseded by the amvalidate checks. http://git.postgresql.org/pg/commitdiff/be44ed27b86ebd165bbedf06a4ac5a8eb943d43c
  • Make extract() do something more reasonable with infinite datetimes. Historically, extract() just returned zero for any case involving an infinite timestamp[tz] input; even cases in which the unit name was invalid. This is not very sensible. Instead, return infinity or -infinity as appropriate when the requested field is one that is monotonically increasing (e.g, year, epoch), or NULL when it is not (e.g., day, hour). Also, throw the expected errors for bad unit names. BACKWARDS INCOMPATIBLE CHANGE Vitaly Burovoy, reviewed by Vik Fearing http://git.postgresql.org/pg/commitdiff/647d87c56ab6da70adb753c08d7cdf7ee905ea8a
  • Remove new coupling between NAMEDATALEN and MAX_LEVENSHTEIN_STRLEN. Commit e529cd4ffa605c6f introduced an Assert requiring NAMEDATALEN to be less than MAX_LEVENSHTEIN_STRLEN, which has been 255 for a long time. Since up to that instant we had always allowed NAMEDATALEN to be substantially more than that, this was ill-advised. It's debatable whether we need MAX_LEVENSHTEIN_STRLEN at all (versus putting a CHECK_FOR_INTERRUPTS into the loop), or whether it has to be so tight; but this patch takes the narrower approach of just not applying the MAX_LEVENSHTEIN_STRLEN limit to calls from the parser. Trusting the parser for this seems reasonable, first because the strings are limited to NAMEDATALEN which is unlikely to be hugely more than 256, and second because the maximum distance is tightly constrained by MAX_FUZZY_DISTANCE (though we'd forgotten to make use of that limit in one place). That means the cost is not really O(mn) but more like O(max(m,n)). Relaxing the limit for user-supplied calls is left for future research; given the lack of complaints to date, it doesn't seem very high priority. In passing, fix confusion between lengths-in-bytes and lengths-in-chars in comments and error messages. Per gripe from Kevin Day; solution suggested by Robert Haas. Back-patch to 9.5 where the unwanted restriction was introduced. http://git.postgresql.org/pg/commitdiff/a396144ac03b0cf337f80201df7e4663cc5a8131
  • Improve levenshtein() docs. Fix chars-vs-bytes confusion here too. Improve poor grammar and markup. http://git.postgresql.org/pg/commitdiff/80aa219146c090d46b599ac40d8d63e30532b622
  • Improve cross-platform consistency of Inf/NaN handling in trig functions. Ensure that the trig functions return NaN for NaN input regardless of what the underlying C library functions might do. Also ensure that an error is thrown for Inf (or otherwise out-of-range) input, except for atan/atan2 which should accept it. All these behaviors should now conform to the POSIX spec; previously, all our popular platforms deviated from that in one case or another. The main remaining platform dependency here is whether the C library might choose to throw a domain error for sin/cos/tan inputs that are large but less than infinity. (Doing so is not unreasonable, since once a single unit-in-the-last-place exceeds PI, there can be no significance at all in the result; however there doesn't seem to be any suggestion in POSIX that such an error is allowed.) We will report such errors if they are reported via "errno", but not if they are reported via "fetestexcept" which is the other mechanism sanctioned by POSIX. Some preliminary experiments with fetestexcept indicated that it might also report errors we could do without, such as complaining about underflow at an unreasonably large threshold. So let's skip that complexity for now. Dean Rasheed, reviewed by Michael Paquier http://git.postgresql.org/pg/commitdiff/fd5200c3dca0bc725f5848eef7ffff538f4479ed
  • Add trigonometric functions that work in degrees. The implementations go to some lengths to deliver exact results for values where an exact result can be expected, such as sind(30) = 0.5 exactly. Dean Rasheed, reviewed by Michael Paquier http://git.postgresql.org/pg/commitdiff/e1bd684a34c11139a1bf4e5200c3bbe59a0fbfad
  • Adjust degree-based trig functions for more portability. The buildfarm isn't very happy with the results of commit e1bd684a34c11139. To try to get the expected exact results everywhere: * Replace M_PI / 180 subexpressions with a precomputed constant, so that the compiler can't decide to rearrange that division with an adjacent operation. Hopefully this will fix failures to get exactly 0.5 from sind(30) and cosd(60). * Add scaling to ensure that tand(45) and cotd(45) give exactly 1; there was nothing particularly guaranteeing that before. * Replace minus zero by zero when tand() or cotd() would output that; many machines did so for tand(180) and cotd(270), but not all. We could alternatively deem both results valid, but that doesn't seem likely to be what users will want. http://git.postgresql.org/pg/commitdiff/73193d82d7c8d849774bf6952dfb4287e213c572
  • Further adjust degree-based trig functions for more portability. The last round didn't do it. Per Noah Misch, the problem on at least some machines is that the compiler pre-evaluates trig functions having constant arguments using code slightly different from what will be used at runtime. Therefore, we must prevent the compiler from seeing constant arguments to any of the libm trig functions used in this code. The method used here might still fail if init_degree_constants() gets inlined into the call sites. That probably won't happen given the large number of call sites; but if it does, we could probably fix it by making init_degree_constants() non-static. I'll avoid that till proven necessary, though. http://git.postgresql.org/pg/commitdiff/65abaab547a5758b0d6d92df4af1663bb47d545f
  • Still further adjust degree-based trig functions for more portability. Indeed, the non-static declaration foreseen in my previous commit message is necessary. Per Noah Misch. http://git.postgresql.org/pg/commitdiff/360f67d31a5656991122b89c9ca22a860f41512c
  • Yet further adjust degree-based trig functions for more portability. Buildfarm member cockatiel is still saying that cosd(60) isn't 0.5. What seems likely is that the subexpression (1.0 - cos(x)) isn't being rounded to double width before more arithmetic is done on it, so force that by storing it into a variable. http://git.postgresql.org/pg/commitdiff/00347575e2754b1aaacd357776560803564d3f35

Tatsuo Ishii pushed:

Andrew Dunstan pushed:

  • Remove Cygwin-specific code from pg_ctl This code has been there for a long time, but it's never really been needed. Cygwin has its own utility for registering, unregistering, stopping and starting Windows services, and that's what's used in the Cygwin postgres packages. So now pg_ctl for Cygwin looks like it is for any Unix platform. Michael Paquier and me http://git.postgresql.org/pg/commitdiff/53c949c1be2f43cd47cb433923e76ea00e9222bc

Ãlvaro Herrera pushed:

Bruce Momjian pushed:

Robert Haas pushed:

  • Support multi-stage aggregation. Aggregate nodes now have two new modes: a "partial" mode where they output the unfinalized transition state, and a "finalize" mode where they accept unfinalized transition states rather than individual values as input. These new modes are not used anywhere yet, but they will be necessary for parallel aggregation. The infrastructure also figures to be useful for cases where we want to aggregate local data and remote data via the FDW interface, and want to bring back partial aggregates from the remote side that can then be combined with locally generated partial aggregates to produce the final value. It may also be useful even when neither FDWs nor parallelism are in play, as explained in the comments in nodeAgg.c. David Rowley and Simon Riggs, reviewed by KaiGai Kohei, Heikki Linnakangas, Haribabu Kommi, and me. http://git.postgresql.org/pg/commitdiff/a7de3dc5c346e07e0439275982569996e645b3c2
  • Support parallel joins, and make related improvements. The core innovation of this patch is the introduction of the concept of a partial path; that is, a path which if executed in parallel will generate a subset of the output rows in each process. Gathering a partial path produces an ordinary (complete) path. This allows us to generate paths for parallel joins by joining a partial path for one side (which at the baserel level is currently always a Partial Seq Scan) to an ordinary path on the other side. This is subject to various restrictions at present, especially that this strategy seems unlikely to be sensible for merge joins, so only nested loops and hash joins paths are generated. This also allows an Append node to be pushed below a Gather node in the case of a partitioned table. Testing revealed that early versions of this patch made poor decisions in some cases, which turned out to be caused by the fact that the original cost model for Parallel Seq Scan wasn't very good. So this patch tries to make some modest improvements in that area. There is much more to be done in the area of generating good parallel plans in all cases, but this seems like a useful step forward. Patch by me, reviewed by Dilip Kumar and Amit Kapila. http://git.postgresql.org/pg/commitdiff/45be99f8cd5d606086e0a458c9c72910ba8a613d

Simon Riggs pushed:

  • Refactor to create generic WAL page read callback. Previously we didn’t have a generic WAL page read callback function, surprisingly. Logical decoding has logical_read_local_xlog_page(), which was actually generic, so move that to xlogfunc.c and rename to read_local_xlog_page(). Maintain logical_read_local_xlog_page() so existing callers still work. As requested by Michael Paquier, Alvaro Herrera and Andres Freund http://git.postgresql.org/pg/commitdiff/422a55a68784fd00f4514834f3649140a9166fa5
  • Speedup 2PC by skipping two phase state files in normal path. 2PC state info is written only to WAL at PREPARE, then read back from WAL at COMMIT PREPARED/ABORT PREPARED. Prepared transactions that live past one bufmgr checkpoint cycle will be written to disk in the same form as previously. Crash recovery path is not altered. Measured performance gains of 50-100% for short 2PC transactions by completely avoiding writing files and fsyncing. Other optimizations still available, further patches in related areas expected. Stas Kelvich and heavily edited by Simon Riggs Based upon earlier ideas and patches by Michael Paquier and Heikki Linnakangas, a concrete example of how Postgres-XC has fed back ideas into PostgreSQL. Reviewed by Michael Paquier, Jeff Janes and Andres Freund Performance testing by Jesper Pedersen http://git.postgresql.org/pg/commitdiff/978b2f65aa1262eb4ecbf8b3785cb1b9cf4db78e
  • Refactor headers to split out standby defs Jeff Janes http://git.postgresql.org/pg/commitdiff/c80b31d557cb4b2d2a65cb0a7e71fd961834fdb2
  • Correct comment in GetConflictingVirtualXIDs() We use Share lock because it is safe to do so. http://git.postgresql.org/pg/commitdiff/1129c2b0ad2732f301f696ae2cf98fb063a4c1f8

Peter Eisentraut pushed:

Fujii Masao pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Alexander Shulgin sent in a patch to fix some IDENTIFICATION divisions.

Dmitry Dolgov sent in another revision of a patch to add array-style subscripting to JSONB.

Tomas Vondra sent in another revision of a patch to add multivariate statistics.

Bruce Momjian and Joe Conway traded patches to expose pg_controldata and pg_config as functions.

Artur Zakirov sent in two more revisions of a patch to implement fuzzy substring searching with the pg_trgm extension.

Ashutosh Bapat sent in two more revisions of a patch to add postgres_fdw join pushdown.

Vitaly Burovoy and Pavel Stěhule traded patches to add a custom function for converting human readable sizes to bytes.

Dilip Kumar sent in another revision of a patch to move PinBuffer and UnpinBuffer to atomics.

Thomas Munro sent in another revision of a patch to add causal reads.

SAWADA Masahiko sent in another revision of a patch to allow multiple synchronous standby servers.

Alexander Shulgin sent in another revision of a patch to add an extension called pg_logical_slot_stream_relation.

Tatsuo Ishii sent in a patch to fix a too-enthusiastic quoting of identifiers with the high bit set.

Anastasia Lubennikova sent in two more revisions of a patch to implement covering unique indexes.

Robert Haas and Etsuro Fujita traded patches to optimize write operations on the PostgreSQL FDW.

Haribabu Kommi sent in two more revisions of a patch to do aggregation in parallel.

David Rowley sent in another revision of a patch to serialize internal aggregate states.

Haribabu Kommi sent in two more revisions of a patch to add combine functions for staged aggregates.

Kyotaro HORIGUCHI sent in another revision of a patch to allow async-capable nodes to register callbacks to run the node before ExecProcNode() and provide an example using same.

Etsuro Fujita sent in another revision of a patch to optimize create_foreignscan_plan/ExecInitForeignScan.

Daniel Verité sent in another revision of a patch to add a \crosstabview to psql.

Tomas Vondra and David Rowley traded patches to optimize outer joins where the outer side is unique.

Petr Jelínek sent in another revision of a patch to enable generic WAL logical messages.

David Rowley sent in a patch to fix an issue where combining aggretates didn't work with pg_dump.

Victor Wagner sent in another revision of a patch to implement failover on the libpq connect level.

Aleksander Alekseev sent in another revision of a patch to optimize dynahashes.

Tomas Vondra sent in another revision of a patch to ensure that more predictable column statistics are being gathered and used.

Fabien COELHO sent in another revision of a patch to add better logging, etc. to pgbench.

David Rowley sent in another revision of a patch to remove functionally dependent GROUP BY columns.

Alexander Korotkov sent in another revision of a patch to add partial sorts.

Pavel Stěhule sent in another revision of a patch to add a parse_ident() function.

Artur Zakirov sent in another revision of a patch to copy regexp_t.

par N Bougain le jeudi 28 janvier 2016 à 20h58

jeudi 21 janvier 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 17 janvier 2016

La LinuxFest Northwest aura lieu les 23 et 24 avril 2016 au Collège Technique de Bellingham (Washington, USA). L'appel à conférenciers est maintenant lancé : http://www.linuxfestnorthwest.org/2016/present

Le PGDay suisse sera, cette année, tenue à l'Université des Sciences Appliquées (HSR) de Rapperswil le 24 juin 2016. L'appel à conférenciers est lancé : http://www.pgday.ch/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Les inscriptions sont encore ouvertes : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016 : http://pgday.in
  • La première conférence PostgreSQL pan-asiatique se tiendra les 16 et 17 mars 2016 à Singapour : http://2016.pgday.asia/
  • Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. Les inscriptions sont encore ouvertes : http://2016.nordicpgday.org/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • FOSS4G NA (Free and Open Source Software for Geospatial - North America) se tiendra à Raleigh, en Caroline du Nord, du 2 au 5 mai 2016. L'appel à conférenciers est lancé : https://2016.foss4g-na.org/cfp
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers est toujours ouvert : http://www.pgcon.org/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20160118004136.GB13585@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Robert Haas pushed:

Tom Lane pushed:

  • Avoid dump/reload problems when using both plpython2 and plpython3. Commit 803716013dc1350f installed a safeguard against loading plpython2 and plpython3 at the same time, but asserted that both could still be used in the same database, just not in the same session. However, that's not actually all that practical because dumping and reloading will fail (since both libraries necessarily get loaded into the restoring session). pg_upgrade is even worse, because it checks for missing libraries by loading every .so library mentioned in the entire installation into one session, so that you can have only one across the whole cluster. We can improve matters by not throwing the error immediately in _PG_init, but only when and if we're asked to do something that requires calling into libpython. This ameliorates both of the above situations, since while execution of CREATE LANGUAGE, CREATE FUNCTION, etc will result in loading plpython, it isn't asked to do anything interesting (at least not if check_function_bodies is off, as it will be during a restore). It's possible that this opens some corner-case holes in which a crash could be provoked with sufficient effort. However, since plpython only exists as an untrusted language, any such crash would require superuser privileges, making it "don't do that" not a security issue. To reduce the hazards in this area, the error is still FATAL when it does get thrown. Per a report from Paul Jones. Back-patch to 9.2, which is as far back as the patch applies without work. (It could be made to work in 9.1, but given the lack of previous complaints, I'm disinclined to expend effort so far back. We've been pretty desultory about support for Python 3 in 9.1 anyway.) http://git.postgresql.org/pg/commitdiff/866566a690bb9916dcd294807e65a6e173396530
  • Use LOAD not actual code execution to pull in plpython library. Commit 866566a690bb9916 is insufficient to prevent dump/reload failures when using transform modules in a database with both plpython2 and plpython3 installed. The reason is that the transform extension scripts use DO blocks as a mechanism to pull in the libpython library before creating the transform function. It's necessary to preload the library because the dynamic loader won't do it for us on every platform, leading to "unresolved symbol" failures when the transform library is loaded. But it's *not* necessary to execute Python code, and doing so will provoke a multiple-Pythons-are-loaded error even after the preceding commit. To fix, use LOAD instead of a DO block. That requires superuser privilege, but creation of a C function does anyway. It also embeds knowledge of the underlying library name for each PL language; but that's wired into the initdb-time contents of pg_pltemplate too, so that doesn't seem like a large problem either. Note that CREATE TRANSFORM as such doesn't call the language module at all. Per a report from Paul Jones. Back-patch to 9.5 where transform modules were introduced. http://git.postgresql.org/pg/commitdiff/fb6fcbd33fbbd6d31fa2b39938e60ecb48dc4de4
  • Remove no-longer-needed old-style check for incompatible plpythons. Commit 866566a690bb9916 introduced a new mechanism for incompatible plpythons to detect each other. I left the old mechanism in place, because it seems possible that a plpython predating that commit might be used with one postdating it. (This would require updating plpython3 but not plpython2 or vice versa, but that seems well within the realm of possibility.) However, surely it will not be able to happen in 9.6 or later, so we can delete the old mechanism in HEAD. http://git.postgresql.org/pg/commitdiff/796d1e889f2b5f88b33a425fdfd08d7906cbd66a
  • Run pgindent on src/bin/pg_dump/* To ease doing indent fixups on a couple of patches I have in progress. http://git.postgresql.org/pg/commitdiff/26905e009babe6020fddcf3820e57e2f87c5539c
  • Access pg_dump's options structs through Archive struct, not directly. Rather than passing around DumpOptions and RestoreOptions as separate arguments, add fields to struct Archive to carry pointers to these objects, and access them through those fields when needed. There already was a RestoreOptions pointer in Archive, though for no obvious reason it was part of the "private" struct rather than out where pg_dump.c could see it. Doing this allows reversion of quite a lot of parameter-addition changes made in commit 0eea8047bf, which is a good thing IMO because this will reduce the code delta between 9.4 and 9.5, probably easing a few future back-patch efforts. Moreover, the previous commit only added a DumpOptions argument to functions that had to have it at the time, which means we could anticipate still more code churn (and more back-patch hazard) as the requirement spread further. I'd hit exactly that problem in my upcoming patch to fix extension membership marking, which is what motivated me to do this. http://git.postgresql.org/pg/commitdiff/5b5fea2a11741e651f7c25e981dd29b610a08426
  • Handle extension members when first setting object dump flags in pg_dump. pg_dump's original approach to handling extension member objects was to run around and clear (or set) their dump flags rather late in its data collection process. Unfortunately, quite a lot of code expects those flags to be valid before that; which was an entirely reasonable expectation before we added extensions. In particular, this explains Karsten Hilbert's recent report of pg_upgrade failing on a database in which an extension has been installed into the pg_catalog schema. Its objects are initially marked as not-to-be-dumped on the strength of their schema, and later we change them to must-dump because we're doing a binary upgrade of their extension; but we've already skipped essential tasks like making associated DO_SHELL_TYPE objects. To fix, collect extension membership data first, and incorporate it in the initial setting of the dump flags, so that those are once again correct from the get-go. This has the undesirable side effect of slightly lengthening the time taken before pg_dump acquires table locks, but testing suggests that the increase in that window is not very much. Along the way, get rid of ugly special-case logic for deciding whether to dump procedural languages, FDWs, and foreign servers; dump decisions for those are now correct up-front, too. In 9.3 and up, this also fixes erroneous logic about when to dump event triggers (basically, they were *always* dumped before). In 9.5 and up, transform objects had that problem too. Since this problem came in with extensions, back-patch to all supported versions. http://git.postgresql.org/pg/commitdiff/e72d7d85310c397a94748db72d73a59c57e0b0dc
  • Fix build_grouping_chain() to not clobber its input lists. There's no good reason for stomping on the input data; it makes the logic in this function no simpler, in fact probably the reverse. And it makes it impossible to separate path generation from plan generation, as I'm working towards doing; that will require more than one traversal of these lists. http://git.postgresql.org/pg/commitdiff/a923af382c5678f3dfb591aacb6b90bf4e5ed7a9
  • Remove dead code in pg_dump. Coverity quite reasonably complained that this check for fout==NULL occurred after we'd already dereferenced fout. However, the check is just dead code since there is no code path by which CreateArchive can return a null pointer. Errors such as can't-open-that-file are reported down inside CreateArchive, and control doesn't return. So let's silence the warning by removing the dead code, rather than continuing to pretend it does something. Coverity didn't complain about this before 5b5fea2a1, so back-patch to 9.5 like that patch. http://git.postgresql.org/pg/commitdiff/57ce9acc04483df4913921d4ff21f01483583fb8

Simon Riggs pushed:

Magnus Hagander pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Simon Riggs sent in another revision of a patch to speed up 2PC.

Marisa Emerson sent in another revision of a patch to support BSD Authentication.

Elvis Pranskevichus sent in a patch to fix an issue with dumping domain constraints in pg_dump.

Michael Paquier and Fabien COELHO traded patches to extend pgbench expressions with functions.

Dilip Kumar sent in another revision of a patch to help make relation extension more scalable.

Etsuro Fujita sent in another revision of a patch to help make FDW DML pushdown more efficient.

SAWADA Masahiko sent in two more revisions of a patch to add a "frozen" bit to the visibility map.

Anastasia Lubennikova sent in two more revisions of a patch to implement covering unique indexes.

Amit Kapila sent in a patch to optimize parallelism code for launched workers usage.

Michael Paquier sent in a patch to remove the service-related code in pg_ctl for Cygwin.

Tom Lane sent in a patch to show what might be needed to fix PL/Python[23] issues in PostgreSQL 9.1. In view of 9.1's short remaining life, this is not going to be applied.

David Rowley sent in two more revisions of a patch to remove functionally dependent GROUP BY columns.

Filip Rembiałkowski sent in a patch to document the fact that BYPASSRLS is a CREATE USER option.

Andres Freund sent in a patch to allow to easily choose between the readiness primitives in unix_latch.c, error out if waiting on socket readiness without a specified socket, only clear unix_latch.c's self-pipe if it actually contains data, and support using epoll as the polling primitive in unix_latch.c.

Amit Langote sent in a patch to fix a comment thinko in expand_inherited_rtentry().

Vinayak Pokale sent in a patch to fix a typo in sequence.c.

Tatsuro Yamada sent in a patch to fix a comment typo in port/atomics/generic.h.

David Rowley sent in two more revisions of a patch to implement combining aggregates.

Mithun C Y sent in a patch to cache snapshot data avoid cuncurrent write to cache.

Etsuro Fujita sent in another revision of a patch to create foreign scan plan.

Etsuro Fujita sent in a patch to update a comment in setrefs.c.

Etsuro Fujita sent in a patch to make a minor documentation tweak to GetForeignPlan documentation.

Alexander Shulgin sent in a POC patch to create pg_logical_slot_stream_relation.

Christian Ullrich sent in another revision of a patch to fix an error in SSPI auth that could cause the wrong realm name to be used.

Jeff Janes sent in another revision of a patch to expose the GIN clean pending list to SQL.

Constantin S. Pan sent in a patch to speed up GIN build with parallel workers.

Andreas Seltenreich sent in a patch to improve spinlock inline assembly for x86.

José Arthur Benetasso Villanova sent in a patch to log operating system user connecting via unix socket.

Dean Rasheed sent in another revision of a patch to implement trigonometric functions in degrees.

par N Bougain le jeudi 21 janvier 2016 à 00h19

mercredi 13 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL - 3

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce dernier billet de la série liste quelques paramètres de configuration qui font leur apparition dans cette nouvelle version.Suivi de l'horodatage des COMMITLe paramètre track_commit_timestamp permet de marquer dans les journaux de transactions ("WAL") chaque validation ("COMMIT") avec la date et l'heure du serveur. Ce paramètre est un booléen, et... Lire Version 9.5 de PostgreSQL - 3

par Sébastien Lardière le mercredi 13 janvier 2016 à 15h12

Rodolphe Quiédeville

Index multi colonnes GIN, GIST

Ce billet intéressera tous les utilisateurs de colonnes de type hstore ou json avec PostgreSQL. Bien que celui-ci prenne pour exemple hstore il s'applique également aux colonnes json ou jsonb.

Commençons par créer une table et remplissons là avec 100 000 lignes de données aléatoires. Notre exemple représente des articles qui sont associés à un identifiant de langue (lang_id) et des tags catégorisés (tags), ici chaque article peut être associé à un pays qui sera la Turquie ou l'Islande.

~# CREATE TABLE article (id int4, lang_id int4, tags hstore);
CREATE TABLE
~# INSERT INTO article 
SELECT generate_series(1,10e4::int4), cast(random()*20 as int),
CASE WHEN random() > 0.5 
THEN 'country=>Turquie'::hstore 
WHEN random() > 0.8 THEN 'country=>Islande' ELSE NULL END AS x;
INSERT 0 100000

Pour une recherche efficace des articles dans une langue donnée nous ajountons un index de type B-tree sur la colonne lang_id et un index de type GIN sur la colonne tags.

~# CREATE INDEX ON article(lang_id);
CREATE INDEX
~# CREATE INDEX ON article USING GIN (tags);
CREATE INDEX

Nous avons maintenant nos données et nos index, nous pouvons commencer les recherches. Recherchons tous les articles écrit en français (on considère que l'id du français est le 17), qui sont associés à un pays (ils ont un tag country), et analysons le plan d'exécution.

~# EXPLAIN ANALYZE SELECT * FROM article WHERE lang_id=17 AND tags ? 'country';
                                                                QUERY PLAN                                                                
------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on article  (cost=122.42..141.21 rows=5 width=35) (actual time=12.348..13.912 rows=3018 loops=1)
   Recheck Cond: ((tags ? 'country'::text) AND (lang_id = 17))
   Heap Blocks: exact=663
   ->  BitmapAnd  (cost=122.42..122.42 rows=5 width=0) (actual time=12.168..12.168 rows=0 loops=1)
         ->  Bitmap Index Scan on article_tags_idx  (cost=0.00..12.75 rows=100 width=0) (actual time=11.218..11.218 rows=60051 loops=1)
               Index Cond: (tags ? 'country'::text)
         ->  Bitmap Index Scan on article_lang_id_idx  (cost=0.00..109.42 rows=4950 width=0) (actual time=0.847..0.847 rows=5016 loops=1)
               Index Cond: (lang_id = 17)
 Planning time: 0.150 ms
 Execution time: 14.111 ms
(10 rows)

On a logiquement 2 parcours d'index, suivis d'une opération de combinaison pour obtenir le résultat final. Pour gagner un peu en performance on penserait naturellement à créer un index multi colonnes qui contienne lang_id et tags, mais si vous avez déjà essayé de le faire vous avez eu ce message d'erreur :

~# CREATE INDEX ON article USING GIN (lang_id, tags);
ERROR:  42704: data type integer has no default operator class for access method "gin"
HINT:  You must specify an operator class for the index or define a default operator class for the data type.
LOCATION:  GetIndexOpClass, indexcmds.c:1246

Le HINT donnne une piste intéressante, en effet les index de type GIN ne peuvent pas s'appliquer sur les colonnes de type int4 (et bien d'autres).

La solution réside dans l'utilisation d'une extension standard, qui combine les opérateurs GIN et B-tree, btree-gin, précisons tout de suite qu'il existe l'équivalent btree-gist.

Comme toute extension elle s'installe aussi simplement que :

~# CREATE EXTENSION btree_gin;
CREATE EXTENSION

Désormais nous allons pouvoir créer notre index multi-colonne et rejouer notre requête pour voir la différence.

~# CREATE INDEX ON article USING GIN (lang_id, tags);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM article WHERE lang_id=17 AND tags ? 'country';
                                                             QUERY PLAN                                                              
-------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on article  (cost=24.05..42.84 rows=5 width=35) (actual time=1.983..3.777 rows=3018 loops=1)
   Recheck Cond: ((lang_id = 17) AND (tags ? 'country'::text))
   Heap Blocks: exact=663
   ->  Bitmap Index Scan on article_lang_id_tags_idx  (cost=0.00..24.05 rows=5 width=0) (actual time=1.875..1.875 rows=3018 loops=1)
         Index Cond: ((lang_id = 17) AND (tags ? 'country'::text))
 Planning time: 0.211 ms
 Execution time: 3.968 ms
(7 rows)

A la lecture de ce deuxième explain le gain est explicite, même avec un petit jeu de données le coût estimé est divisé par 3, l'on gagne une lecture d'index et une opération de composition. Maintenant nous pouvons supprimer les 2 autres index pour ne conserver que celui-ci.

par Rodolphe Quiédeville le mercredi 13 janvier 2016 à 14h11

mardi 12 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL - 2

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce deuxième billet évoque donc les nouvelles méthodes internes du moteur, c'est-à-dire de nouveaux outils dont PostgreSQL dispose pour traiter les données.Index BRINIl s'agit d'un nouveau type d'index, créé pour résoudre des problèmes d'accès à de très gros volumes de données ; le cas d'usage est une table de log, dans laquelle les données sont... Lire Version 9.5 de PostgreSQL - 2

par Sébastien Lardière le mardi 12 janvier 2016 à 10h45

lundi 11 janvier 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 10 janvier 2016

PostgreSQL 9.5 est disponible ! http://www.postgresql.org/docs/current/static/release-9-5.html https://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.5
[ndt: version francophone : http://blog.postgresql.fr/index.php?post/PostgreSQL-9-5]

FOSS4G NA (Free and Open Source Software for Geospatial - North America) se tiendra à Raleigh, en Caroline du Nord, du 2 au 5 mai 2016. L'appel à conférenciers est lancé : https://2016.foss4g-na.org/cfp

PostgreSQL@SCaLE aura lieu les 21 & 22 janvier 2016 au centre des conventions de Pasadena : https://reg.socallinuxexpo.org/reg6/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Les inscriptions sont encore ouvertes : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016. L'appel à conférenciers est lancé : http://pgday.in
  • La première conférence PostgreSQL pan-asiatique se tiendra les 16 et 17 mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/
  • Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. Les inscriptions sont encore ouvertes : http://2016.nordicpgday.org/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers a été lancé : http://www.pgcon.org/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20160110232921.GA3645@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Do some copy-editing on the docs for row-level security. Clarifications, markup improvements, corrections of misleading or outright wrong statements. http://git.postgresql.org/pg/commitdiff/c1611db01fec587525e88270854c4b993846dcb3
  • Fix bogus lock release in RemovePolicyById and RemoveRoleFromObjectPolicy. Can't release the AccessExclusiveLock on the target table until commit. Otherwise there is a race condition whereby other backends might service our cache invalidation signals before they can actually see the updated catalog rows. Just to add insult to injury, RemovePolicyById was closing the rel (with incorrect lock drop) and then passing the now-dangling rel pointer to CacheInvalidateRelcache. Probably the reason this doesn't fall over on CLOBBER_CACHE buildfarm members is that some outer level of the DROP logic is still holding the rel open ... but it'd have bit us on the arse eventually, no doubt. http://git.postgresql.org/pg/commitdiff/f47b602df80d7647ca2e71c86f7228b1bf5bf9f3
  • Fix regrole and regnamespace types to honor quoting like other reg* types. Aside from any consistency arguments, this is logically necessary because the I/O functions for these types also handle numeric OID values. Without a quoting rule it is impossible to distinguish numeric OIDs from role or namespace names that happen to contain only digits. Also change the to_regrole and to_regnamespace functions to dequote their arguments. While not logically essential, this seems like a good idea since the other to_reg* functions do it. Anyone who really wants raw lookup of an uninterpreted name can fall back on the time-honored solution of (SELECT oid FROM pg_namespace WHERE nspname = whatever). Report and patch by Jim Nasby, reviewed by Michael Paquier http://git.postgresql.org/pg/commitdiff/fb1227af67eae5e97795f7e3563673c6e67d2844
  • Fix regrole and regnamespace output functions to do quoting, too. We discussed this but somehow failed to implement it... http://git.postgresql.org/pg/commitdiff/b0cadc08fea564f75a0702e15b2bd949377bd2f3
  • Adjust behavior of row_security GUC to match the docs. Some time back we agreed that row_security=off should not be a way to bypass RLS entirely, but only a way to get an error if it was being applied. However, the code failed to act that way for table owners. Per discussion, this is a must-fix bug for 9.5.0. Adjust the logic in rls.c to behave as expected; also, modify the error message to be more consistent with the new interpretation. The regression tests need minor corrections as well. Also update the comments about row_security in ddl.sgml to be correct. (The official description of the GUC in config.sgml is already correct.) I failed to resist the temptation to do some other very minor cleanup as well, such as getting rid of a duplicate extern declaration. http://git.postgresql.org/pg/commitdiff/5d35438273c4523a4dc4b48c3bd575e64310d3d4
  • Docs: provide a concrete discussion and example for RLS race conditions. Commit 43cd468cf01007f3 added some wording to create_policy.sgml purporting to warn users against a race condition of the sort that had been noted some time ago by Peter Geoghegan. However, that warning was far too vague to be useful (or at least, I completely failed to grasp what it was on about). Since the problem case occurs with a security design pattern that lots of people are likely to try to use, we need to be as clear as possible about it. Provide a concrete example in the main-line docs in place of the original warning. http://git.postgresql.org/pg/commitdiff/7debf36072b3a088b3003aab6dcf57c3f186100d
  • In psql's tab completion, change most TailMatches patterns to Matches. In the refactoring in commit d37b816dc9e8f976c8913296781e08cbd45c5af1, we mostly kept to the original design whereby only the last few words on the line were matched to identify a completable pattern. However, after commit d854118c8df8c413d069f7e88bb01b9e18e4c8ed, there's really no reason to do it like that: where it's sensible, we can use patterns that expect to match the entire input line. And mostly, it's sensible. Matching the entire line greatly reduces the odds of a false match that leads to offering irrelevant completions. Moreover (though I've not tried to measure this), it should make tab completion faster since many of the patterns will be discarded after a single integer comparison that finds that the wrong number of words appear on the line. There are certain identifiable places where we still need to use TailMatches because the statement in question is allowed to appear embedded in a larger statement. These are just a small minority of the existing patterns, though, so the benefit of switching where possible is large. It's possible that this patch has removed some within-line matching behaviors that are in fact desirable, but we can put those back when we get complaints. Most of the removed behaviors are certainly silly. Michael Paquier, with some further adjustments by me http://git.postgresql.org/pg/commitdiff/9b181b0363deb65b15a9feaf3eb74f86707498a9
  • Convert psql's tab completion for backslash commands to the new style. This requires adding some more infrastructure to handle both case-sensitive and case-insensitive matching, as well as the ability to match a prefix of a previous word. So it ends up being about a wash line-count-wise, but it's just as big a readability win here as in the SQL tab completion rules. Michael Paquier, some adjustments by me http://git.postgresql.org/pg/commitdiff/4f18010af126f126824e01eec2285e6263d98b3d
  • Add to_regnamespace() and to_regrole() to the documentation. Commits cb9fa802b32b222b and 0c90f6769de6a60f added these functions, but did not bother with documentation. http://git.postgresql.org/pg/commitdiff/83be1844acdcb0cbff31369a65ec61d588fbe9f3
  • Make the to_reg*() functions accept text not cstring. Using cstring as the input type was a poor decision, because that's not really a full-fledged type. In particular, it lacks implicit coercions from text or varchar, meaning that usages like to_regproc('foo'||'bar') wouldn't work; basically the only case that did work without explicit casting was a simple literal constant argument. The lack of field complaints about this suggests that hardly anyone is using these functions, so hopefully fixing it won't cause much of a compatibility problem. They've only been there since 9.4, anyway. Petr Korobeinikov http://git.postgresql.org/pg/commitdiff/ea0d494dae0d3d6fce26bf5d6fbaa07e2ee6c402
  • In opr_sanity regression test, check for unexpected uses of cstring. In light of commit ea0d494dae0d3d6f, it seems like a good idea to add a regression test that will complain about random functions taking or returning cstring. Only I/O support functions and encoding conversion functions should be declared that way. While at it, add some checks that encoding conversion functions are declared properly. Since pg_conversion isn't populated manually, it's not quite as necessary to check its contents as it is for catalogs like pg_proc; but one thing we definitely have not tested in the past is whether the identified conproc for a conversion actually does that conversion vs. some other one. http://git.postgresql.org/pg/commitdiff/921191912c48a68db81c02c02f3bc22e291d918c
  • Sort $(wildcard) output where needed for reproducible build output. The order of inclusion of .o files makes a difference in linker output; not a functional difference, but still a bitwise difference, which annoys some packagers who would like reproducible builds. Report and patch by Christoph Berg http://git.postgresql.org/pg/commitdiff/3343ea9e8ea4f552b3f6e5436938f2f0e153b947
  • Remove some ancient and unmaintained encoding-conversion test cruft. In commit 921191912c48a68d I claimed that we weren't testing encoding conversion functions, but further poking around reveals that we did have an equivalent though hard-wired set of tests in conversion.sql. AFAICS there is no advantage to doing it like that as compared to letting the catalog contents drive the test, so let the opr_sanity addition stand and remove the now-redundant tests in conversion.sql. Also, remove some infrastructure in src/backend/utils/mb/conversion_procs for building conversion.sql's list of tests. That was unmaintained, and had not corresponded to the actual contents of conversion.sql since 2007 or perhaps even further back. http://git.postgresql.org/pg/commitdiff/419400c5da738d86c87e903a3d1924ff365bf203
  • Comment typo fix. Per Amit Langote. http://git.postgresql.org/pg/commitdiff/4bf87169cc1890442aa694f3057e0a0ad60c51f4
  • In initdb's post-bootstrap phase, drop temp tables explicitly. Although these temp tables will get removed from template1 at the end of the standalone-backend run, that's too late to keep them from getting copied into the template0 and postgres databases, now that we use only a single backend run for the whole sequence. While no real harm is done by the extra copies (since they'd be deleted on first use of the temp schema), it's still unsightly, and it would mean some wasted cycles for every database creation for the life of the installation. Oversight in commit c4a8812cf64b1426. Noticed by Amit Langote. http://git.postgresql.org/pg/commitdiff/dad08994b25b8cd2caa83b2e856fcc940d5e515c
  • Provide more detail in postmaster log for password authentication failures. We tell people to examine the postmaster log if they're unsure why they are getting auth failures, but actually only a few relatively-uncommon failure cases were given their own log detail messages in commit 64e43c59b817a78d. Expand on that so that every failure case detected within md5_crypt_verify gets a specific log detail message. This should cover pretty much every ordinary password auth failure cause. So far I've not noticed user demand for a similar level of auth detail for the other auth methods, but sooner or later somebody might want to work on them. This is not that patch, though. http://git.postgresql.org/pg/commitdiff/5e0b5dcab685fe2a342385450a29a825cf40cddf
  • Remove vestigial CHECK_FOR_INTERRUPTS call. Commit e710b65c inserted code in md5_crypt_verify to disable and later re-enable interrupts, with a CHECK_FOR_INTERRUPTS call as part of the second step, to process any interrupts that had been held off. Commit 6647248e removed the interrupt disable/re-enable code, but left behind the CHECK_FOR_INTERRUPTS, even though this is now an entirely random, pointless place for one. md5_crypt_verify doesn't run long enough to need such a check, and if it did, this would still be the wrong place to put one. http://git.postgresql.org/pg/commitdiff/6b1a837f69d00d265bee4b57ba2d320f1463f131
  • Use plain mkdir() not pg_mkdir_p() to create subdirectories of PGDATA. When we're creating subdirectories of PGDATA during initdb, we know darn well that the parent directory exists (or should exist) and that the new subdirectory doesn't (or shouldn't). There is therefore no need to use anything more complicated than mkdir(). Using pg_mkdir_p() just opens us up to unexpected failure modes, such as the one exhibited in bug #13853 from Nuri Boardman. It's not very clear why pg_mkdir_p() went wrong there, but it is clear that we didn't need to be trying to create parent directories in the first place. We're not even saving any code, as proven by the fact that this patch nets out at minus five lines. Since this is a response to a field bug report, back-patch to all branches. http://git.postgresql.org/pg/commitdiff/33b054bc797628e418e379badd38b00e4b523115
  • Fix unobvious interaction between -X switch and subdirectory creation. Turns out the only reason initdb -X worked is that pg_mkdir_p won't whine if you point it at something that's a symlink to a directory. Otherwise, the attempt to create pg_xlog/ just like all the other subdirectories would have failed. Let's be a little more explicit about what's happening. Oversight in my patch for bug #13853 (mea culpa for not testing -X ...) http://git.postgresql.org/pg/commitdiff/b41fb65056076b42d64a8690d61fd73dc648645b
  • Delay creation of subplan tlist until after create_plan(). Once upon a time it was necessary for grouping_planner() to determine the tlist it wanted from the scan/join plan subtree before it called query_planner(), because query_planner() would actually make a Plan using that. But we refactored things a long time ago to delay construction of the Plan tree till later, so there's no need to build that tlist until (and indeed unless) we're ready to plaster it onto the Plan. The only thing query_planner() cares about is what Vars are going to be needed for the tlist, and it can perfectly well get that by looking at the real tlist rather than some masticated version. Well, actually, there is one minor glitch in that argument, which is that make_subplanTargetList also adds Vars appearing only in HAVING to the tlist it produces. So now we have to account for HAVING explicitly in build_base_rel_tlists. But that just adds a few lines of code, and I doubt it moves the needle much on processing time; we might be doing pull_var_clause() twice on the havingQual, but before we had it scanning dummy tlist entries instead. This is a very small down payment on rationalizing grouping_planner enough so it can be refactored. http://git.postgresql.org/pg/commitdiff/c44d013835049053d19bc1795f0d169f3d1d6ff0
  • Marginal cleanup of GROUPING SETS code in grouping_planner(). Improve comments and make it a shade less messy. I think we might want to move all of this somewhere else later, but it needs to be more readable first. In passing, re-pgindent the file, affecting some recently-added comments concerning parallel query planning. http://git.postgresql.org/pg/commitdiff/a54676acadcf811f6945db15e81651df96beabc4
  • Add STRICT to some C functions created by the regression tests. These functions readily crash when passed a NULL input value. The tests themselves do not pass NULL values to them; but when the regression database is used as a basis for fuzz testing, they cause a lot of noise. Also, if someone were to leave a regression database lying about in a production installation, these would create a minor security hazard. Andreas Seltenreich http://git.postgresql.org/pg/commitdiff/529baf6a2f3fe85e7e6b4ad3ca38ed4ebffd6bb4
  • Clean up code for widget_in() and widget_out(). Given syntactically wrong input, widget_in() could call atof() with an indeterminate pointer argument, typically leading to a crash; or if it didn't do that, it might return a NULL pointer, which again would lead to a crash since old-style C functions aren't supposed to do things that way. Fix that by correcting the off-by-one syntax test and throwing a proper error rather than just returning NULL. Also, since widget_in and widget_out have been marked STRICT for a long time, their tests for null inputs are just dead code; remove 'em. In the oldest branches, also improve widget_out to use snprintf not sprintf, just to be sure. In passing, get rid of a long-since-useless sprintf into a local buffer that nothing further is done with, and make some other minor coding style cleanups. In the intended regression-testing usage of these functions, none of this is very significant; but if the regression test database were left around in a production installation, these bugs could amount to a minor security hazard. Piotr Stefaniak, Michael Paquier, and Tom Lane http://git.postgresql.org/pg/commitdiff/1cb63c791c7d070c1bb3cce58885c9697d769cd2
  • Clean up some lack-of-STRICT issues in the core code, too. A scan for missed proisstrict markings in the core code turned up these functions: brin_summarize_new_values pg_stat_reset_single_table_counters pg_stat_reset_single_function_counters pg_create_logical_replication_slot pg_create_physical_replication_slot pg_drop_replication_slot The first three of these take OID, so a null argument will normally look like a zero to them, resulting in "ERROR: could not open relation with OID 0" for brin_summarize_new_values, and no action for the pg_stat_reset_XXX functions. The other three will dump core on a null argument, though this is mitigated by the fact that they won't do so until after checking that the caller is superuser or has rolreplication privilege. In addition, the pg_logical_slot_get/peek[_binary]_changes family was intentionally marked nonstrict, but failed to make nullness checks on all the arguments; so again a null-pointer-dereference crash is possible but only for superusers and rolreplication users. Add the missing ARGISNULL checks to the latter functions, and mark the former functions as strict in pg_proc. Make that change in the back branches too, even though we can't force initdb there, just so that installations initdb'd in future won't have the issue. Since none of these bugs rise to the level of security issues (and indeed the pg_stat_reset_XXX functions hardly misbehave at all), it seems sufficient to do this. In addition, fix some order-of-operations oddities in the slot_get_changes family, mostly cosmetic, but not the part that moves the function's last few operations into the PG_TRY block. As it stood, there was significant risk for an error to exit without clearing historical information from the system caches. The slot_get_changes bugs go back to 9.4 where that code was introduced. Back-patch appropriate subsets of the pg_proc changes into all active branches, as well. http://git.postgresql.org/pg/commitdiff/26d538dc93543ed80c315b8313ea4dacd7309ff6
  • Add some checks on "char"-type columns to type_sanity and opr_sanity. I noticed that the sanity checks in the regression tests omitted to check a couple of "poor man's enum" columns that you'd reasonably expect them to check. There are other "char"-type columns in system catalogs that are not covered by either type_sanity or opr_sanity, e.g. pg_rewrite.ev_type. However, those catalogs are not populated with any manually-created data during bootstrap, so it seems less necessary to check them this way. http://git.postgresql.org/pg/commitdiff/3ef16c46fb3a64c150a3b42c3cc4a8538a12ff3f
  • Remove a useless PG_GETARG_DATUM() call from jsonb_build_array. This loop uselessly fetched the argument after the one it's currently looking at. No real harm is done since we couldn't possibly fetch off the end of memory, but it's confusing to the reader. Also remove a duplicate (and therefore confusing) PG_ARGISNULL check in jsonb_build_object. I happened to notice these things while trolling for missed null-arg checks earlier today. Back-patch to 9.5, not because there is any real bug, but just because 9.5 and HEAD are still in sync in this file and we might as well keep them so. In passing, re-pgindent. http://git.postgresql.org/pg/commitdiff/820bdccc1be22513e1aaa441d554992a5a2e314f

Robert Haas pushed:

Ãlvaro Herrera pushed:

  • Make pg_shseclabel available in early backend startup. While the in-core authentication mechanism doesn't need to access pg_shseclabel at all, it's reasonable to think that an authentication hook will want to look at the label for the role logging in, or for rows in other catalogs used during the authentication phase of startup. Catalog version bumped, because this changes the "is nailed" status for pg_shseclabel. Author: Adam Brightwell http://git.postgresql.org/pg/commitdiff/efa318bcfac132c48dff8196f726e56a6843f06b
  • Make pg_receivexlog silent with 9.3 and older servers. A pointless and confusing error message is shown to the user when attempting to identify a 9.3 or older remote server with a 9.5/9.6 pg_receivexlog, because the return signature of IDENTIFY_SYSTEM was changed in 9.4. There's no good reason for the warning message, so shuffle code around to keep it quiet. (pg_recvlogical is also affected by this commit, but since it obviously cannot work with 9.3 that doesn't actually matter much.) Backpatch to 9.5. Reported by Marco Nenciarini, who also wrote the initial patch. Further tweaked by Robert Haas and Fujii Masao; reviewed by Michael Paquier and Craig Ringer. http://git.postgresql.org/pg/commitdiff/4aecd22d3c84c44dd230426bcccd286798ac6b65
  • Add scale(numeric) Author: Marko Tiikkaja http://git.postgresql.org/pg/commitdiff/abb1733922f3ff17a514499883a549f8bd03af44
  • Windows: Make pg_ctl reliably detect service status. pg_ctl is using isatty() to verify whether the process is running in a terminal, and if not it sends its output to Windows' Event Log ... which does the wrong thing when the output has been redirected to a pipe, as reported in bug #13592. To fix, make pg_ctl use the code we already have to detect service-ness: in the master branch, move src/backend/port/win32/security.c to src/port (with suitable tweaks so that it runs properly in backend and frontend environments); pg_ctl already has access to pgport so it Just Works. In older branches, that's likely to cause trouble, so instead duplicate the required code in pg_ctl.c. Author: Michael Paquier Bug report and diagnosis: Egon Kocjan Backpatch: all supported branches http://git.postgresql.org/pg/commitdiff/a967613911f7ef7b6387b9e8718f0ab8f0c4d9c8
  • pgstat: add WAL receiver status view & SRF. This new view provides insight into the state of a running WAL receiver in a HOT standby node. The information returned includes the PID of the WAL receiver process, its status (stopped, starting, streaming, etc), start LSN and TLI, last received LSN and TLI, timestamp of last message send and receipt, latest end-of-WAL LSN and time, and the name of the slot (if any). Access to the detailed data is only granted to superusers; others only get the PID. Author: Michael Paquier Reviewer: Haribabu Kommi http://git.postgresql.org/pg/commitdiff/b1a9bad9e744857291c7d5516080527da8219854
  • Add win32security to LIBOBJS. This seems to fix Mingw's compile that was broken in a967613911f7e, as evidenced by buildfarm. http://git.postgresql.org/pg/commitdiff/fa838b555f90039ae5f0e6fb86ccae6a88b42703
  • Fix order of arguments to va_start() http://git.postgresql.org/pg/commitdiff/f81c966d2095fdab70a5d81ceb6dd9c89f4acd87
  • Blind attempt at a Cygwin fix. Further portability fix for a967613911f7. Mingw- and MSVC-based builds appear to be working fine, but Cygwin needs an extra tweak whereby the new win32security.c file is explicitely added to the list of files to build in pgport, per Cygwin members brolga and lorikeet. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/e9282e953205a2f3125fc8d1052bc01cb77cd2a3
  • Revert "Blind attempt at a Cygwin fix." This reverts commit e9282e953205a2f3125fc8d1052bc01cb77cd2a3, which blew up in a pretty spectacular way. Re-introduce the original code while we search for a real fix. http://git.postgresql.org/pg/commitdiff/463172116634423f8708ad9d7afb0f759a40cf2c

Tatsuo Ishii pushed:

Magnus Hagander pushed:

Simon Riggs pushed:

  • Avoid pin scan for replay of XLOG_BTREE_VACUUM. Replay of XLOG_BTREE_VACUUM during Hot Standby was previously thought to require complex interlocking that matched the requirements on the master. This required an O(N) operation that became a significant problem with large indexes, causing replication delays of seconds or in some cases minutes while the XLOG_BTREE_VACUUM was replayed. This commit skips the “pin scan†that was previously required, by observing in detail when and how it is safe to do so, with full documentation. The pin scan is skipped only in replay; the VACUUM code path on master is not touched here. The current commit still performs the pin scan for toast indexes, though this can also be avoided if we recheck scans on toast indexes. Later patch will address this. No tests included. Manual tests using an additional patch to view WAL records and their timing have shown the change in WAL records and their handling has successfully reduced replication delay. http://git.postgresql.org/pg/commitdiff/687f2cd7a0150647794efe432ae0397cb41b60ff
  • Revoke change to rmgr desc of btree vacuum. Per discussion with Andres Freund http://git.postgresql.org/pg/commitdiff/b6028426137532afae00188405fdecf7057b208c

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Ian Lawrence Barwick sent in a patch to update the description of vacuumdb to fit current realities.

Haribabu Kommi sent in four more revisions of a patch to implement multi-tenancy with RLS.

Pavel Stěhule sent in another revision of a patch to implement num_notnulls().

Pavel Stěhule sent in three more revisions of a patch to implement pg_size_bytes().

Michael Paquier sent in another revision of a patch to ensure that dynloader.h be present in Windows installations.

Amul Sul sent in a patch to ensure that columns in tables made inheritance tables by ALTER TABLE ... INHERIT are actually dropped when they're dropped in the parents.

Vitaly Burovoy sent in another revision of a patch to allow non-crazy extraction of fields from 'infinity'::TIMESTAMP[TZ].

Alexander Shulgin sent in a patch to fix an inconsistency in error handling for the START_REPLICATION command.

Petr Korobeinikov sent in a patch to add schema-qualified relnames in constraint error messages.

Alexander Shulgin sent in a patch to add an \errverbose option to psql to include schema-qualified names in error messages.

Amit Kapila sent in two more revisions of a patch to refactor LWLock tranches.

Andreas Karlsson sent in another revision of a patch to add COPY (statement) tab completion to psql.

Amit Kapila sent in another revision of a patch to rename pgproc variables.

Andreas Karlsson sent in another revision of a patch to improve tab completion in psql for FDW DDL.

Kyotaro HORIGUCHI sent in another revision of a patch to prepare for sharing psqlscan with pgbench, change the access method to shell variables, detach common.c from psqlscan, ensure that pgbench uses a common frontend SQL parser, and change the way to hold command list.

Ashutosh Bapat sent in another revision of a patch to enable getting sorted data from foreign server for merge joins.

Ashutosh Bapat sent in a patch to remove duplicate parameter deparser code from postgres_fdw.

Craig Ringer sent in another revision of a patch to implement pg_logical_output.

Marisa Emerson sent in two revisions of a patch to add support for BSD authentication, used on OpenBSD.

Etsuro Fujita sent in a patch to fix an issue with updating foreign tables.

Peter Geoghegan sent in another revision of a patch to generalize SortSupport for text.

Konstantin Knizhnik sent in two revisions of a patch to optimize LIMIT clauses.

Craig Ringer sent in another revision of a patch to implement pglogical.

David Rowley sent in another revision of a patch to implement combining aggregates.

Ashutosh Bapat sent in a patch to fix some infelicities in FDW join pushdown and scanclauses.

Ãlvaro Herrera sent in another revision of a patch to allow cataloging NOT NULL constraints.

Tomas Vondra sent in another revision of a patch to extend the HyperLogLog API.

Stas Kelvich and Simon Riggs traded patches to speed up two-phase transactions.

Peter Geoghegan sent in a patch to fix some misspellings in nodeGather.c and controldata.c.

Michael Paquier sent in another revision of a patch to fix some issues with Cygwin.

Peter Eisentraut sent in a patch to fix some odd bugs in psql's tab completion of CREATE INDEX.

Christian Ullrich sent in another revision of a patch to fix an SSPI auth bug.

Christian Ullrich sent in a patch to close a handle leak in SSPI auth.

par N Bougain le lundi 11 janvier 2016 à 21h51

vendredi 8 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce premier billet revient donc sur les ajouts au langage SQL de la version 9.5 de PostgreSQL.Ces ajouts portent sur de nombreux champs de fonctionnalités, de la sécurité aux gestions de performances en passant par la gestion des transactions.UPSERTCe mot clé désigne en réalité la possibilité d'intercepter une erreur de clé primaire sur un ordre... Lire Version 9.5 de PostgreSQL

par Sébastien Lardière le vendredi 8 janvier 2016 à 10h49

jeudi 7 janvier 2016

Actualités PostgreSQL.fr

PostgreSQL 9.5 : UPSERT, Row Level Security et Big Data

Le 7 janvier 2016 : Le PostgreSQL Global Development Group annonce la publication de la version 9.5 de PostgreSQL. Cette version ajoute les capacités d'UPSERT, les droits de niveau ligne (Row Level Security), et de nombreuses fonctionnalités orientées Big Data, qui ouvrent plus grand encore les possibilités d'utilisation de PostgreSQL. Avec ces nouvelles fonctionnalités, PostgreSQL va devenir le choix par défaut pour un nombre encore plus grands d'applications, qu'il s'agisse de startups, de grandes entreprises ou d'agences gouvernementales.

Annie Prévot, DSI de la CNAF, déclare, « La CNAF sert 11 millions d'allocataires. Elle verse 73 milliards d'euros par an, au travers de 26 types de prestations. Ce service, essentiel à la population, s'appuie sur un système d'information qui se doit d'être absolument performant et fiable. Le système d'information de la CNAF s'appuie avec satisfaction sur le gestionnaire de Base de données PostgreSQL. En utilisation sur la moitié des systèmes, ce logiciel libre offre toutes les garanties de fonctionnement et de fiabilité. Il est en phase de déploiement sur l'ensemble des systèmes. »

UPSERT

UPSERT, raccourci de « INSERT, ON CONFLICT UPDATE », est une fonctionnalité réclamée depuis longtemps par les développeurs d'applications. Elle permet de traiter de la même façon l'ajout ou l'actualisation de lignes. UPSERT simplifie le développement des applications web et mobile en permettant de laisser la base gérer les conflits entre modifications concurrentes des données. Cette fonctionnalité supprime également la dernière barrière à la migration d'applications MySQL historiques vers PostgreSQL.

Développée sur les deux dernières années par Peter Geoghegan, programmeur au sein de la société Heroku, l'implantation de UPSERT au sein de PostgreSQL est nettement plus souple et puissante que celle de la plupart des autres SGBDR. La nouvelle clause ON CONFLICT permet d'ignorer la nouvelle donnée, ou d'actualiser différentes colonnes ou relations, de façon à supporter les chaînes d'ETL (Extract, Transform, LOAD) les plus complexes. Et, à l'instar de PostgreSQL, cette fonctionnalité a été conçue pour être totalement concurrentielle, et s'intégrer avec les autres fonctionnalités de PostgreSQL, dont la réplication logique.

Row Level Security

PostgreSQL continue d'étendre les possibilités de sécurisation des accès, avec la nouvelle fonctionnalité Row Level Security (RLS). RLS propose un vrai contrôle d'accès par ligne et par colonne, qui s'intègre avec les outils externe de sécurisation, tel SELinux. PostgreSQL est déjà connu comme « la base la plus sécurisée par défaut ». RLS conforte cette position de meilleur choix par défaut pour les applications à fort besoin de sécurisation, telles que la conformité PCI, la dIrective Européenne de protection des données, et les standards de protection des données des systèmes de santé.

RLS est l'aboutissement de 5 ans d'ajout de fonctionnalités de sécurité à PostgreSQL, ce qui inclut les travaux de KaiGai Kohei de NEC, Stephen Frost de Crunchy Data, et ceux de Dean Rasheed. À travers cette fonctionnalité, les administrateurs de base de données peuvent définir des politiques de sécurité, qui filtrent les lignes visibles en fonction des utilisateurs. La sécurisation mises en place par ce biais est résistante aux injections SQL et autres trous de sécurité de niveau applicatif.

Fonctionnalités Big Data

PostgreSQL 9.5 inclut de nombreuses nouvelles fonctionnalités pour les bases plus volumineuses, et l'intégration avec les systèmes Big Data. Ces fonctionnalités garantissent que PostgreSQL va continuer de jouer un rôle majeur dans le marché grandissant du Big Data en logiciels libres. Citons :

Les index BRIN : Ce nouveau type d'index permet la création d'index beaucoup plus petits, mais très efficaces, pour les tables volumineuses. Il est nécessaire que les données soient ordonnées. Par exemple, des tables contenant des millions de lignes de traces peuvent être indexées et interrogées en 5% du temps nécessaire lors de l'utilisation d'index de type BTree.

Tris plus rapides : PostgreSQL trie désormais les données textuelles et numériques plus rapidement, à l'aide d'un algorithme appelé "clés abrégées" ou "abbreviated keys". Cela accélère certaines requêtes qui nécessitent de trier des volumes considérables de données d'un facteur 2 à 12, et peut également accélérer la création d'index d'un facteur 20.

CUBE, ROLLUP et GROUPING SETS : Ces nouvelles clauses du standard permettent de produire des rapports avec plusieurs niveaux de synthèse en une seule requête. CUBE permet également une meilleure intégration de PostrgreSQL avec un nombre plus grand d'outils de création de rapports OLAP (Online Analytic Processing), tel Tableau.

Les Foreign Data Wrappers (FDW) : Ils permettent déjà d'utiliser PostgreSQL pour interroger des systèmes Big Data, tel Hadoop ou Cassandra. La version 9.5 ajoute IMPORT FOREIGN SCHEMA et les jointures au niveau des sources externes, ce qui rend les connexions aux données externes plus efficaces et plus faciles à établir.

TABLESAMPLE : Cette clause SQL permet de récupérer rapidement un échantillon statistique d'une table sans tri coûteux.

« Le nouvel index BRIN apparu dans PostgreSQL 9.5 est une fonctionnalité puissante qui permet à PostgreSQL de gérer et indexer des volumes de données difficiles, voire impossibles à traiter par le passé. Il permet de faire évoluer les données et les performances au delà de la limite considérée précédemment avec les bases relationnelles traditionnelles et fait de PostgreSQL une solution parfaite pour les analyses Big Data," déclare Boyan Botev, Lead Database Administrator, Premier, Inc.

Liens

par daamien le jeudi 7 janvier 2016 à 16h12

mercredi 6 janvier 2016

Actualités PostgreSQL.fr

PG Day France 2016 à Lille : Appel à Orateurs

Le PGDay France est la conférence annuelle de la communauté francophone de PostgreSQL. Cette année, l’événement se tiendra le 31 mai à Lille. Une centaine de visiteurs sont attendus pour une journée d'échanges autour de PostgreSQL et de ses projets associés. Retrouvez plus d'informations sur le site de l’événement : http://www.pgday.fr

Vous êtes expert sur un domaine lié aux bases de données libres ? Vous avez utilisé PostgreSQL dans un contexte spécifique (gros volumes, forte charge, client reconnu, projet innovant, etc.) ? Vous participez à un projet libre lié à PostgreSQL ? Alors n'hésitez pas à proposer une présentation !

Pour l’édition 2016, les thèmes particulièrement mis en lumière sont les suivants :

  • Administration de bases volumineuses
  • Études de cas / témoignages
  • Industrialisation
  • Entrepôts de données
  • Systèmes décisionnels
  • Travaux sur la sémantique
  • Big Data
  • Data Mining / Exploration de Données
  • Systèmes d'Information Géographiques

Cette liste n'est pas exhaustive. Il est possible de proposer d'autres sujets liés à PostgreSQL.

La conférence PGDay France est à destination des professionnels, notamment les directeurs informatiques, les décideurs, les chefs de projets, les administrateurs de bases de données, les développeurs, les administrateurs systèmes et tous les profils qui entrent en contact avec un SGBD.

Pour soumettre une intervention, il vous suffit d'envoyer un e-mail à l'adresse contact@pgday.fr, en précisant les éléments suivants :

  • vos nom et prénom
  • votre société/employeur
  • une bio succincte (300 caractères max.)
  • votre compte twitter (optionnel)
  • le titre de votre intervention
  • la durée de votre intervention (45 min. max.)
  • une description courte (200 caractères max.)
  • une description longue (700 caractères max.)
  • une photo (200x200 pixels minimum)

Les interventions devront être en français et disponibles sous licence libre. Les interventions pourront faire l'objet d'une captation audio/vidéo et d'une diffusion sur internet.

La date limite de réception des propositions est fixée au 14 février 2016 à 23h59 CEST.

Dans le courant du mois de février 2016, un sondage communautaire sera organisé au sein de la communauté francophone pour évaluer les différentes propositions.

Ensuite, le comité de sélection étudiera toutes les propositions valides. Le choix des sessions sera basé sur la présentation de la soumission, son intérêt pour une audience professionnelle, la cohérence du programme de la journée et sur le résultat du vote préliminaire.

La décision du comité de sélection sera finale et sans appel.

Les membres du comité s'expriment en leur nom propre. Leurs choix ne reflètent pas la position de leur employeur.

Les orateurs sélectionnés seront avertis par e-mail avant le 7 mars 2016.

Pour toute question à propos de cet appel à conférenciers et du PGDay France en général, vous pouvez envoyer un message à l'adresse : contact@pgday.fr

par daamien le mercredi 6 janvier 2016 à 17h03

mardi 5 janvier 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 3 janvier 2016

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Détails et appel à conférenciers ci-après : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016. L'appel à conférenciers est lancé : http://pgday.in
  • La première conférence PostgreSQL pan-asiatique se tiendra les 16 et 17 mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/
  • Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. L'appel à conférenciers est lancé : http://2016.nordicpgday.org/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers a été lancé : http://www.pgcon.org/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20160104004346.GH30007@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Ãlvaro Herrera pushed:

Tom Lane pushed:

  • Update documentation about pseudo-types. Tone down an overly strong statement about which pseudo-types PLs are likely to allow. Add "event_trigger" to the list, as well as "pg_ddl_command" in 9.5/HEAD. Back-patch to 9.3 where event_trigger was added. http://git.postgresql.org/pg/commitdiff/731dfc7d5f07fac2c3c72f47c29a947e363acee9
  • Fix omission of -X (--no-psqlrc) in some psql invocations. As of commit d5563d7df, psql -c no longer implies -X, but not all of our regression testing scripts had gotten that memo. To ensure consistency of results across different developers, make sure that *all* invocations of psql in all scripts in our tree use -X, even where this is not what previously happened. Michael Paquier and Tom Lane http://git.postgresql.org/pg/commitdiff/870df2b3b77414a536d6533566628f11f8f309ec
  • Document the exponentiation operator as associating left to right. Common mathematical convention is that exponentiation associates right to left. We aren't going to change the parser for this, but we could note it in the operator's description. (It's already noted in the operator precedence/associativity table, but users might not look there.) Per bug #13829 from Henrik Pauli. http://git.postgresql.org/pg/commitdiff/54aaafe95f65c95fd9ba085826af87d778c94613
  • Code and docs review for cube kNN support. Commit 33bd250f6c4cc309f4eeb657da80f1e7743b3e5c could have done with some more review: Adjust coding so that compilers unfamiliar with elog/ereport don't complain about uninitialized values. Fix misuse of PG_GETARG_INT16 to retrieve arguments declared as "integer" at the SQL level. (This was evidently copied from cube_ll_coord and cube_ur_coord, but those were wrong too.) Fix non-style-guide-conforming error messages. Fix underparenthesized if statements, which pgindent would have made a hash of, and remove some unnecessary parens elsewhere. Run pgindent over new code. Revise documentation: repeated accretion of more operators without any rethinking of the text already there had left things in a bit of a mess. Merge all the cube operators into one table and adjust surrounding text appropriately. David Rowley and Tom Lane http://git.postgresql.org/pg/commitdiff/81ee726d87ec67c4f2846110c99f72e8a20dcd07
  • Put back one copyObject() in rewriteTargetView(). Commit 6f8cb1e23485bd6d tried to centralize rewriteTargetView's copying of a target view's Query struct. However, it ignored the fact that the jointree->quals field was used twice. This only accidentally failed to fail immediately because the same ChangeVarNodes mutation is applied in both cases, so that we end up with logically identical expression trees for both uses (and, as the code stands, the second ChangeVarNodes call actually does nothing). However, we end up linking *physically* identical expression trees into both an RTE's securityQuals list and the WithCheckOption list. That's pretty dangerous, mainly because prepsecurity.c is utterly cavalier about further munging such structures without copying them first. There may be no live bug in HEAD as a consequence of the fact that we apply preprocess_expression in between here and prepsecurity.c, and that will make a copy of the tree anyway. Or it may just be that the regression tests happen to not trip over it. (I noticed this only because things fell over pretty badly when I tried to relocate the planner's call of expand_security_quals to before expression preprocessing.) In any case it's very fragile because if anyone tried to make the securityQuals and WithCheckOption trees diverge before we reach preprocess_expression, it would not work. The fact that the current code will preprocess securityQuals and WithCheckOptions lists at completely different times in different query levels does nothing to increase my trust that that can't happen. In view of the fact that 9.5.0 is almost upon us and the aforesaid commit has seen exactly zero field testing, the prudent course is to make an extra copy of the quals so that the behavior is not different from what has been in the field during beta. http://git.postgresql.org/pg/commitdiff/fd1952575618cacf7afa544d8b89ddb77be9eaee
  • Add some comments about division of labor between rewriter and planner. The rationale for the way targetlist processing is done wasn't clearly stated anywhere, and I for one had forgotten some of the details. Having just painfully re-learned them, add some breadcrumbs for the next person. http://git.postgresql.org/pg/commitdiff/efe4c9d7049f0bf832b792bfad05c92aaf86aa3c
  • Minor hacking on contrib/cube documentation. Improve markup, particularly of the table of functions; add or improve examples for some of the functions; wordsmith some of the function descriptions. http://git.postgresql.org/pg/commitdiff/e5e5267a91f4880c121bf50865cbc38078441989
  • Avoid useless truncation attempts during VACUUM. VACUUM can skip heap pages altogether when there's a run of consecutive pages that are all-visible according to the visibility map. This causes it to not update its nonempty_pages count, just as if those pages were empty, which means that at the end we will think they are candidates for deletion. Thus, we may take the table's AccessExclusive lock only to find that no pages are really truncatable. This usually causes no real problems on a master server, thanks to the lock being acquired only conditionally; but on hot-standby servers, the same lock must be acquired unconditionally which can result in unnecessary query cancellations. To improve matters, force examination of the table's last page whenever we reach there with a nonempty_pages count that would allow a truncation attempt. If it's not empty, we'll advance nonempty_pages and thereby prevent the truncation attempt. If we are unable to acquire cleanup lock on that page, there's no need to force it, unless we're doing an anti-wraparound vacuum. We can just check for tuples with a shared buffer lock and then give up. (When we are doing an anti-wraparound vacuum, and decide it's okay to skip the page because it contains no freezable tuples, this patch still improves matters because nonempty_pages is properly updated, which it was not before.) Since only the last page is special-cased in this way, we might attempt a truncation that will release many fewer pages than the normal heuristic would suggest; at worst, only one page would be truncated. But that seems all right, because the situation won't repeat during the next vacuum. The real problem with the old logic is that the useless truncation attempt happens every time we vacuum, so long as the state of the last few dozen pages doesn't change. This is a longstanding deficiency, but since the consequences aren't very severe in most scenarios, I'm not going to risk a back-patch. Jeff Janes and Tom Lane http://git.postgresql.org/pg/commitdiff/e842908233bb9c5cea0e765fc828b52badd8228e
  • Dept of second thoughts: the !scan_all exit mustn't increase scanned_pages. In the extreme edge case where contended pages are the only ones that escape being scanned, the previous commit would have allowed us to think that relfrozenxid could be advanced, which is exactly wrong. http://git.postgresql.org/pg/commitdiff/e5d06f2b12a7c75f2b0c7fd2055a14efaa2b59ec
  • Fix ALTER OPERATOR to update dependencies properly. Fix an oversight in commit 321eed5f0f7563a0: replacing an operator's selectivity functions needs to result in a corresponding update in pg_depend. We have a function that can handle that, but it was not called by AlterOperator(). To fix this without enlarging pg_operator.h's #include list beyond what clients can safely include, split off the function definitions into a new file pg_operator_fn.h, similarly to what we've done for some other catalog header files. It's not entirely clear whether any client-side code needs to include pg_operator.h, but it seems prudent to assume that there is some such code somewhere. http://git.postgresql.org/pg/commitdiff/0dab5ef39b3d9d86e45bbbb2f6ea60b4f5517d9a
  • Add a comment noting that FDWs don't have to implement EXCEPT or LIMIT TO. postgresImportForeignSchema pays attention to IMPORT's EXCEPT and LIMIT TO options, but only as an efficiency hack, not for correctness' sake. The FDW documentation does explain that, but someone using postgres_fdw.c as a coding guide might not remember it, so let's add a comment here. Per question from Regina Obe. http://git.postgresql.org/pg/commitdiff/5f36096b77fe47015cbac130d1a20d089f202a1e
  • Add some more defenses against silly estimates to gincostestimate(). A report from Andy Colson showed that gincostestimate() was not being nearly paranoid enough about whether to believe the statistics it finds in the index metapage. The problem is that the metapage stats (other than the pending-pages count) are only updated by VACUUM, and in the worst case could still reflect the index's original empty state even when it has grown to many entries. We attempted to deal with that by scaling up the stats to match the current index size, but if nEntries is zero then scaling it up still gives zero. Moreover, the proportion of pages that are entry pages vs. data pages vs. pending pages is unlikely to be estimated very well by scaling if the index is now orders of magnitude larger than before. We can improve matters by expanding the use of the rule-of-thumb estimates I introduced in commit 7fb008c5ee59b040: if the index has grown by more than a cutoff amount (here set at 4X growth) since VACUUM, then use the rule-of-thumb numbers instead of scaling. This might not be exactly right but it seems much less likely to produce insane estimates. I also improved both the scaling estimate and the rule-of-thumb estimate to account for numPendingPages, since it's reasonable to expect that that is accurate in any case, and certainly pages that are in the pending list are not either entry or data pages. As a somewhat separate issue, adjust the estimation equations that are concerned with extra fetches for partial-match searches. These equations suppose that a fraction partialEntries / numEntries of the entry and data pages will be visited as a consequence of a partial-match search. Now, it's physically impossible for that fraction to exceed one, but our estimate of partialEntries is mostly bunk, and our estimate of numEntries isn't exactly gospel either, so we could arrive at a silly value. In the example presented by Andy we were coming out with a value of 100, leading to insane cost estimates. Clamp the fraction to one to avoid that. Like the previous patch, back-patch to all supported branches; this problem can be demonstrated in one form or another in all of them. http://git.postgresql.org/pg/commitdiff/3c93a60f6019768f5742b7893a93db93fb67e71f
  • Teach flatten_reloptions() to quote option values safely. flatten_reloptions() supposed that it didn't really need to do anything beyond inserting commas between reloption array elements. However, in principle the value of a reloption could be nearly anything, since the grammar allows a quoted string there. Any restrictions on it would come from validity checking appropriate to the particular option, if any. A reloption value that isn't a simple identifier or number could thus lead to dump/reload failures due to syntax errors in CREATE statements issued by pg_dump. We've gotten away with not worrying about this so far with the core-supported reloptions, but extensions might allow reloption values that cause trouble, as in bug #13840 from Kouhei Sutou. To fix, split the reloption array elements explicitly, and then convert any value that doesn't look like a safe identifier to a string literal. (The details of the quoting rule could be debated, but this way is safe and requires little code.) While we're at it, also quote reloption names if they're not safe identifiers; that may not be a likely problem in the field, but we might as well try to be bulletproof here. It's been like this for a long time, so back-patch to all supported branches. Kouhei Sutou, adjusted some by me http://git.postgresql.org/pg/commitdiff/c7e27becd2e6eb93b20965b9f22701eaad42a764
  • Update copyright for 2016 Manually fix some copyright lines missed by the automated script. http://git.postgresql.org/pg/commitdiff/ad08bf5c8b96c2a3a70d96f5be1c04cb83b4ed6e
  • Update copyright for 2016 On closer inspection, the reason copyright.pl was missing files is that it is looking for 'Copyright (c)' and they had 'Copyright (C)'. Fix that, and update a couple more that grepping for that revealed. http://git.postgresql.org/pg/commitdiff/48c9f2889a4ad25a771d13b88f2778a306f2d970
  • Make copyright.pl cope with nonstandard case choices in copyright notices. The need for this is shown by the files it missed in Bruce's recent run. I fixed it so that it will actually adjust the case when needed. In passing, also make it skip .po files, since those will just get overwritten anyway from the translation repository. http://git.postgresql.org/pg/commitdiff/de7c8dbea1a17a0e1709c4b12371615d28e21c13
  • Adjust back-branch release note description of commits a2a718b22 et al. As pointed out by Michael Paquier, recovery_min_apply_delay didn't exist in 9.0-9.3, making the release note text not very useful. Instead make it talk about recovery_target_xid, which did exist then. 9.0 is already out of support, but we can fix the text in the newer branches' copies of its release notes. http://git.postgresql.org/pg/commitdiff/df35af2ca7b5545d32b978a88b665bac2b9fa638
  • Fix overly-strict assertions in spgtextproc.c. spg_text_inner_consistent is capable of reconstructing an empty string to pass down to the next index level; this happens if we have an empty string coming in, no prefix, and a dummy node label. (In practice, what is needed to trigger that is insertion of a whole bunch of empty-string values.) Then, we will arrive at the next level with in->level == 0 and a non-NULL (but zero length) in->reconstructedValue, which is valid but the Assert tests weren't expecting it. Per report from Andreas Seltenreich. This has no impact in non-Assert builds, so should not be a problem in production, but back-patch to all affected branches anyway. In passing, remove a couple of useless variable initializations and shorten the code by not duplicating DatumGetPointer() calls. http://git.postgresql.org/pg/commitdiff/7157fe80f42476db249e062b4f6eef6a3994b234
  • Teach pg_dump to quote reloption values safely. Commit c7e27becd2e6eb93 fixed this on the backend side, but we neglected the fact that several code paths in pg_dump were printing reloptions values that had not gotten massaged by ruleutils. Apply essentially the same quoting logic in those places, too. http://git.postgresql.org/pg/commitdiff/b416c0bb622ce5d33fdbec3bbce00451132f10ec
  • Fix treatment of *lpNumberOfBytesRecvd == 0: that's a completion condition. pgwin32_recv() has treated a non-error return of zero bytes from WSARecv() as being a reason to block ever since the current implementation was introduced in commit a4c40f140d23cefb. However, so far as one can tell from Microsoft's documentation, that is just wrong: what it means is graceful connection closure (in stream protocols) or receipt of a zero-length message (in message protocols), and neither case should result in blocking here. The only reason the code worked at all was that control then fell into the retry loop, which did *not* treat zero bytes specially, so we'd get out after only wasting some cycles. But as of 9.5 we do not normally reach the retry loop and so the bug is exposed, as reported by Shay Rojansky and diagnosed by Andres Freund. Remove the unnecessary test on the byte count, and rearrange the code in the retry loop so that it looks identical to the initial sequence. Back-patch to 9.5. The code is wrong all the way back, AFAICS, but since it's relatively harmless in earlier branches we'll leave it alone. http://git.postgresql.org/pg/commitdiff/90e61df8130dc7051a108ada1219fb0680cb3eb6
  • Do a final round of copy-editing on the 9.5 release notes. http://git.postgresql.org/pg/commitdiff/027989197aab9e555328721b003ebd1839a16704
  • Do some copy-editing on the docs for replication origins. Minor grammar and markup improvements. http://git.postgresql.org/pg/commitdiff/c6aeba353a15d71f584488a7482fb88337f843e3
  • Guard against null arguments in binary_upgrade_create_empty_extension(). The CHECK_IS_BINARY_UPGRADE macro is not sufficient security protection if we're going to dereference pass-by-reference arguments before it. But in any case we really need to explicitly check PG_ARGISNULL for all the arguments of a non-strict function, not only the ones we expect null values for. Oversight in commits 30982be4e5019684e1772dd9170aaa53f5a8e894 and f92fc4c95ddcc25978354a8248d3df22269201bc. Found by Andreas Seltenreich. (The other usages in pg_upgrade_support.c seem safe.) http://git.postgresql.org/pg/commitdiff/939d10cd8711fdeb7f0ff62c9c6b08e3eddbba3e

Joe Conway pushed:

  • Rename (new|old)estCommitTs to (new|old)estCommitTsXid. The variables newestCommitTs and oldestCommitTs sound as if they are timestamps, but in fact they are the transaction Ids that correspond to the newest and oldest timestamps rather than the actual timestamps. Rename these variables to reflect that they are actually xids: to wit newestCommitTsXid and oldestCommitTsXid respectively. Also modify related code in a similar fashion, particularly the user facing output emitted by pg_controldata and pg_resetxlog. Complaint and patch by me, review by Tom Lane and Alvaro Herrera. Backpatch to 9.5 where these variables were first introduced. http://git.postgresql.org/pg/commitdiff/241448b23adf3432988f2b4012ff90a338b4d0bf

Peter Eisentraut pushed:

Noah Misch pushed:

  • Fix comments about WAL rule "write xlog before data" versus pg_multixact. Recovery does not achieve its goal of zeroing all pg_multixact entries whose accompanying WAL records never reached disk. Remove that claim and justify its expendability. Detail the need for TrimMultiXact(), which has little in common with the TrimCLOG() rationale. Merge two tightly-related comments. Stop presenting pg_multixact as specific to heap_lock_tuple(); PostgreSQL 9.3 extended its use to heap_update(). Noticed while investigating a report from Andres Freund. http://git.postgresql.org/pg/commitdiff/3cd1ba147e5619199914e5b71e0edbd188a763d2
  • Cover heap_page_prune_opt()'s cleanup lock tactic in README. Jeff Janes, reviewed by Jim Nasby. http://git.postgresql.org/pg/commitdiff/dfcd9cb30237f882b7308bdcbfb0318b22b1e224

Bruce Momjian pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in another revision of a patch to use quicksort when performing external sorts.

Jeff Janes sent in another revision of a patch to add a gin_clean_pending_list() function.

David Rowley sent in two more revisions of a patch to fix some typos.

Grzegorz Sampolski sent in a patch to add rhost as an option for PAM auth.

Haribabu Kommi and Alexander Shulgin traded patches to add a pg_hba_lookup() function.

Dmitry Ivanov sent in a patch to add an array_elemtype() function.

Aleksander Alekseev sent in three more revisions of a patch to fix lock contention for HASHHDR.mutex.

Andreas Karlsson sent in a patch to implement tab completion in psql for COPY with a query.

Joe Conway sent in another revision of a patch to expose pg_controldata and pg_config as functions.

Haribabu Kommi sent in another revision of a patch to implement multi-tenancy with RLS.

Thomas Munro sent in another revision of a patch to implement causal reads.

Tom Lane sent in a patch to better detail logging for password auth failures.

Michael Paquier sent in three revisions of a patch to add support for a case-sensitive comparison facility to psql's tab completion.

Bruce Momjian sent in a patch to ensure that the correct .h files are installed on Windows.

Amit Kapila sent in two more revisions of a patch to refactor LWLock tranches.

Andreas Karlsson sent in a patch to improve psql's tab completion for FDW DDL.

Stas Kelvich sent in another revision of a patch to add Tsvector editing functions.

Pavel Stěhule sent in another revision of a patch to add a custom function for converting human-readable sizes to bytes.

Ãlvaro Herrera sent in another revision of a patch to implement column stores.

Corey Huinker sent in another revision of a patch to allow FETCH to be limited by a specification measured in bytes rather than rows.

Tom Lane sent in a patch to improve the rows estimate for BRIN indexes.

Pavel Stěhule sent in another revision of a patch to make a PL/PythonU version of ereport().

Dilip Kumar sent in another revision of a patch to scale relation extensibility.

David Steele sent in a patch to add a pg_audit extension.

Tomas Vondra sent in another revision of a patch to postpone building buckets to the end of Hash (in HashJoin).

Tomas Vondra sent in another revision of a patch to add a Bloom filter in Hash Joins with batches.

Tomas Vondra sent in a patch to extend the hyperloglog API by adding initHyperLogLogError() and freeHyperLogLog().

Petr Jelínek sent in a patch to add pglogical, a logical replication contrib module.

Petr Jelínek sent in another revision of a patch to add a sequence access method.

Petr Jelínek sent in a patch to fix a copy-paste error in the logical decoding docs.

Petr Jelínek sent in a patch to implement generic WAL logical messages.

Simon Riggs sent in a WIP patch to implement failover slots.

Michael Paquier sent in another revision of a patch to fix how pg_dump locks tables.

Jim Nasby sent in two revisions of a patch to improve error reporting in format().

Amit Kapila and Andres Freund traded patches to fix an issue with backend processes not terminating properly.

SAWADA Masahiko sent in another revision of a patch to support N synchronous standby servers for N>1.

Simon Riggs sent in another revision of a patch to avoid standby pin scans.

Andreas Karlsson sent in two approaches to a patch to fix some infelicities between psql's \x auto feature and EXPLAIN.

Pavel Stěhule sent in another revision of a patch to implement a num_nulls() function.

par N Bougain le mardi 5 janvier 2016 à 20h45

vendredi 1 janvier 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 27 décembre 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Détails et appel à conférenciers ci-après : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016. L'appel à conférenciers est lancé : http://pgday.in
  • La première conférence PostgreSQL pan-asiatique se tiendra les 16 et 17 mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/
  • Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. L'appel à conférenciers est lancé : http://2016.nordicpgday.org/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers a été lancé : http://www.pgcon.org/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151228022000.GA6856@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Stephen Frost pushed:

  • Make viewquery a copy in rewriteTargetView() Rather than expect the Query returned by get_view_query() to be read-only and then copy bits and pieces of it out, simply copy the entire structure when we get it. This addresses an issue where AcquireRewriteLocks, which is called by acquireLocksOnSubLinks(), scribbles on the parsetree passed in, which was actually an entry in relcache, leading to segfaults with certain view definitions. This also future-proofs us a bit for anyone adding more code to this path. The acquireLocksOnSubLinks() was added in commit c3e0ddd40. Back-patch to 9.3 as that commit was. http://git.postgresql.org/pg/commitdiff/6f8cb1e23485bd6d45e8865760436e1a5ce65a6d

Tom Lane pushed:

  • Fix calculation of space needed for parsed words in tab completion. Yesterday in commit d854118c8, I had a serious brain fade leading me to underestimate the number of words that the tab-completion logic could divide a line into. On input such as "(((((", each character will get seen as a separate word, which means we do indeed sometimes need more space for the words than for the original line. Fix that. http://git.postgresql.org/pg/commitdiff/f5a4370aea3580f5f7f59a77e41fde62f2be12d8
  • Allow omitting one or both boundaries in an array slice specifier. Omitted boundaries represent the upper or lower limit of the corresponding array subscript. This allows simpler specification of many common use-cases. (Revised version of commit 9246af6799819847faa33baf441251003acbb8fe) YUriy Zhuravlev http://git.postgresql.org/pg/commitdiff/6efbded6e4672c597a6f0dc0f09263e7db7369ff
  • In pg_dump, remember connection passwords no matter how we got them. When pg_dump prompts the user for a password, it remembers the password for possible re-use by parallel worker processes. However, libpq might have extracted the password from a connection string originally passed as "dbname". Since we don't record the original form of dbname but break it down to host/port/etc, the password gets lost. Fix that by retrieving the actual password from the PGconn. (It strikes me that this whole approach is rather broken, as it will also lose other information such as options that might have been present in the connection string. But we'll leave that problem for another day.) In passing, get rid of rather silly use of malloc() for small fixed-size arrays. Back-patch to 9.3 where parallel pg_dump was introduced. Report and fix by Zeus Kronion, adjusted a bit by Michael Paquier and me http://git.postgresql.org/pg/commitdiff/1aa41e3eae3746e05d0e23286ac740a9a6cee7df
  • Improve handling of password reuse in src/bin/scripts programs. This reverts most of commit 83dec5a71 in favor of having connectDatabase() store the possibly-reusable password in a static variable, similar to the coding we've had for a long time in pg_dump's version of that function. To avoid possible problems with unwanted password reuse, make callers specify whether it's reasonable to attempt to re-use the password. This is a wash for cases where re-use isn't needed, but it is far simpler for callers that do want that. Functionally there should be no difference. Even though we're past RC1, it seems like a good idea to back-patch this into 9.5, like the prior commit. Otherwise, if there are any third-party users of connectDatabase(), they'll have to deal with an API change in 9.5 and then another one in 9.6. Michael Paquier http://git.postgresql.org/pg/commitdiff/ff402ae11b4d33e0e46c2730f63033d3631b8010
  • Avoid VACUUM FULL altogether in initdb. Commit ed7b3b3811c5836a purported to remove initdb's use of VACUUM FULL, as had been agreed to in a pghackers discussion back in Dec 2014. But it missed this one ... http://git.postgresql.org/pg/commitdiff/01e386a325549b7755739f31308de4be8eea110d
  • Fix factual and grammatical errors in comments for struct _tableInfo. Amit Langote, further adjusted by me http://git.postgresql.org/pg/commitdiff/96cd61a16958d3a64da697c3ef31eee5e10141a0
  • Docs typo fix. Michael Paquier http://git.postgresql.org/pg/commitdiff/bee172fcd586bccd3a3ba067592d639b7600aa04
  • Docs: fix erroneously-given function name. pg_replication_session_is_setup() exists nowhere; apparently this is meant to refer to pg_replication_origin_session_is_setup(). Adrien Nayrat http://git.postgresql.org/pg/commitdiff/71dd092c0177af14a00bbb18a8aebbed0d389f05
  • Remove unnecessary row ordering dependency in pg_rewind test suite. t/002_databases.pl was expecting to see a specific physical order of the rows in pg_database. I broke that in HEAD with commit 01e386a325549b77, but I'd say it's a pretty fragile test methodology in any case, so fix it in 9.5 as well. http://git.postgresql.org/pg/commitdiff/a9246fbf665327870370d1088bfc9efdfd2719ec
  • Fix brin_summarize_new_values() to check index type and ownership. brin_summarize_new_values() did not check that the passed OID was for an index at all, much less that it was a BRIN index, and would fail in obscure ways if it wasn't (possibly damaging data first?). It also lacked any permissions test; by analogy to VACUUM, we should only allow the table's owner to summarize. Noted by Jeff Janes, fix by Michael Paquier and me http://git.postgresql.org/pg/commitdiff/3d2b31e30e2931b3edb5ab9d0eafca13e7bcffe5
  • Include typmod when complaining about inherited column type mismatches. MergeAttributes() rejects cases where columns to be merged have the same type but different typmod, which is correct; but the error message it printed didn't show either typmod, which is unhelpful. Changing this requires using format_type_with_typemod() in place of TypeNameToString(), which will have some minor side effects on the way some type names are printed, but on balance this is an improvement: the old code sometimes printed one type according to one set of rules and the other type according to the other set, which could be confusing in its own way. Oddly, there were no regression test cases covering any of this behavior, so add some. Complaint and fix by Amit Langote http://git.postgresql.org/pg/commitdiff/fec1ad94dfc5ddacfda8d249bf4b3c739da8f7a1

Robert Haas pushed:

  • postgres_fdw: Consider requesting sorted data so we can do a merge join. When use_remote_estimate is enabled, consider adding ORDER BY to the query we sending to the remote server so that we can use that ordered data for a merge join. Commit f18c944b6137329ac4a6b2dce5745c5dc21a8578 arranges to push down the query pathkeys, which seems like the case mostly likely to be a win, but testing shows this can sometimes win, too. For a regular table, we know which indexes are present and therefore test whether the ordering provided by each such index is useful. Here, we take the opposite approach: guess what orderings would be useful if they could be generated cheaply, and then ask the remote side what those will cost. Ashutosh Bapat, with very substantial cosmetic revisions by me. Also reviewed by Rushabh Lathia. http://git.postgresql.org/pg/commitdiff/ccd8f97922944566d26c7d90eb67ab7848ee9905
  • Comment improvements for abbreviated keys. Peter Geoghegan and Robert Haas http://git.postgresql.org/pg/commitdiff/0ba3f3bc65f1176250b942e14fd9e4975a5d3913
  • Change Gather not to use a physical tlist. This should have been part of the original commit, but was missed. Pushing data between processes is expensive, so we definitely want to project away unneeded columns here, just as we do for other nodes like Sort and Hash that care about the volume of data. http://git.postgresql.org/pg/commitdiff/51d152f18e124cc07c293756cc16014ba218b2df
  • Read from the same worker repeatedly until it returns no tuple. The original coding read tuples from workers in round-robin fashion, but performance testing shows that it works much better to read enough to empty one queue before moving on to the next. I believe the reason for this is that, with the old approach, we could easily wake up a worker repeatedly to write only one new tuple into the shm_mq each time. With this approach, by the time the process gets scheduled, it has a decent chance of being able to fill the entire buffer in one go. Patch by me. Dilip Kumar helped with performance testing. http://git.postgresql.org/pg/commitdiff/bc7fcab5e36b9597857fa7e3fa6d9ba54aaea167

Peter Eisentraut pushed:

Teodor Sigaev pushed:

Fujii Masao pushed:

Ãlvaro Herrera pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Michael Paquier sent in another revision of a patch to fix bug #13685.

Michael Paquier sent in another revision of a patch to add in-core regression tests for replication, cascading, archiving, PITR, etc.

Haribabu Kommi sent in another revision of a patch to implement parallel aggregation.

Amit Kapila sent in another revision of a patch to speed up clog access by increasing CLOG buffers.

Simon Riggs sent in a patch to avoid the "pin scan" on standbys.

Victor Wagner sent in another revision of a patch to implement failover on libpq connect level.

Artur Zakirov and Pavel Stěhule traded patches to fix %TYPE and add %ARRAYTYPE and %ELEMENTTYPE to PL/pgsql.

Stephen Frost sent in two more revisions of a patch to add a note regarding permissions in pg_catalog, reserve the "pg_" namespace for roles, and create default roles.

Etsuro Fujita sent in two more revisions of a patch to optimize writes to the PostgreSQL FDW.

Kyotaro HORIGUCHI sent in another revision of a patch to fix some issues in psql's tab completion for CREATE/DROP INDEX.

Pavel Stěhule sent in another revision of a patch to add a pg_size_bytes() function.

Noah Misch sent in another revision of a patch to speed up writing out stats.

Aleksander Alekseev sent in another revision of a patch to fix lock contention for HASHHDR.mutex.

YUriy Zhuravlev sent in another revision of a patch to extend the array slice syntax.

Tomas Vondra sent in another revision of a patch to add multivariate statistics.

Robert Haas sent in another revision of a patch to add parallel joins.

Haribabu Kommi sent in another revision of a patch to add a pg_hba_lookup function to get all matching pg_hba.conf entries.

Peter Geoghegan sent in a patch to test a maximum order for external sort runs.

Michael Paquier sent in a patch to fix a typo in the pg_rewind docs.

David Rowley sent in another revision of a patch to make it possible to combine aggregates. This is infrastructure for, among other things, making these easier to parallelize and shard.

Ildus Kurbangaliev sent in three more revisions of a patch to add GiST support for UUIDs.

Daniel Verité sent in another revision of a patch to add a \crosstabview command to psql.

Aleksander Alekseev sent in two more revisions of a patch to improve performance of tables with many partitions.

Ildus Kurbangaliev sent in another revision of a patch to refactor lwlock tranches.

Alexander Korotkov sent in another revision of a patch to rework the access method interface.

SAWADA Masahiko sent in a patch to allow sending multiple options to ALTER ROLE...(RE)SET

Jeff Janes sent in another revision of a patch to eliminate spurious standby query cancellations.

Emre Hasegeli sent in another revision of a patch to add a BRIN correlation cost estimate.

SAWADA Masahiko sent in another revision of a patch to support N (N > 1) synchronous standby servers.

Vinayak Pokale sent in another revision of a patch to implement a VACUUM progress checker.

Teodor Sigaev sent in a patch to support OR clauses in index scans.

Joe Conway sent in another revision of a patch to expose pg_controldata and pg_config as functions.

Jeff Janes sent in a patch to avoid endless futile table locks in vacuuming.

Artur Zakirov sent in another revision of a patch to add fuzzy substring searching to the pg_trgm extension.

Peter Geoghegan sent in a patch to make some fixes and changes to the INSERT documentation.

par N Bougain le vendredi 1 janvier 2016 à 17h12

mercredi 30 décembre 2015

Nicolas Thauvin

Réflexions sur l'archivage des fichiers WAL

PostgreSQL faisant son bonhomme de chemin, on se retrouve désormais avec des configurations où il faut archiver plusieurs fois les WAL parce qu'on a de la sauvegarde PITR et de la réplication.

Le plus gros piège lorsqu'on a du PITR et de la réplication, ou plusieurs serveurs standby, c'est d'oublier que chaque élément de l'architecture qui consomme du WAL doit avoir son propre répertoire de WAL archivés, car chacun purge les fichiers différemment.

On tombe dans le piège facilement, en se disant, "pas de problème pour la purge des vieux fichiers, c'est le PITR ou le slave le plus éloigné qui purgera". Si la purge du PITR passe alors que le standby était déconnecté du maitre, cette purge peut casser la réplication.

La solution est d'archiver plusieurs fois le même fichier WAL. Pour optimiser, le plus efficace à l'usage est d'utiliser des hardlinks. En gros, on archive une fois le fichier WAL et on crée autant de lien hard qu'il faut pour les autres consommateurs de WAL archivés. Rappelons, que la donnée n'est supprimée que lorsqu'il n'existe plus aucun lien et que plus aucun processus n'a le fichier ouvert, à ne pas confondre avec un lien symbolique.

Pour archiver vite, il vaut mieux éviter de compresser et stocker les archives soit en local, soit sur un partage NFS, l'archivage par SSH restant le plus lent. Tout est compromis entre chiffrage des communications sur le réseau et espace disque disponible, les liens hard restant rapides à créer et avec une consommation d'espace disque supplémentaire négligeable.

Enfin, PostgreSQL exécute la commande d'archivage avec l'appel system() qui fork un shell : toutes les possibilités du shell sont alors disponibles, par exemple :

archive_command = 'rsync -a %p slave1:/archived_xlog/slave1/%f && ssh slave1 "for h in slave2 pitr; do ln /archived_xlog/slave1/%f /archived_xlog/$h/%f; done"'

Oui, une boucle et une seule copie avec rsync par SSH pour 3 utilisations. On préfèrera surement faire un script pour rendre les choses plus lisibles. Ça marche aussi pour restore_command et archive_cleanup_command dans recovery.conf :

restore_command = 'cp /archived_xlog/$(hostname)/%f %p'
archive_cleanup_command = 'pg_archivecleanup /archived_xlog/$(hostname) %r'

mercredi 30 décembre 2015 à 21h01

lundi 21 décembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 20 décembre 2015

PostgreSQL 9.5RC1 disponible. À vos tests ! https://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.5

Le PGday annuel indien se tiendra à Bangalore (État du Karnataka en Inde) le 26 février 2016. L'appel à conférenciers est lancé : http://pgday.in

Le PGDay nordique, une série de conférences sur une seule journée, aura lieu à Helsinki (Finlande) le 17 mars 2016. L'appel à conférenciers est lancé : http://2016.nordicpgday.org/

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Détails et appel à conférenciers ci-après : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • La première conférence PostgreSQL pan-asiatique se tiendra en mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers a été lancé : http://www.pgcon.org/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151221041252.GB16788@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Andres Freund pushed:

  • Correct statement to actually be the intended assert statement. e3f4cfc7 introduced a LWLockHeldByMe() call, without the corresponding Assert() surrounding it. Spotted by Coverity. Backpatch: 9.1+, like the previous commit http://git.postgresql.org/pg/commitdiff/2a3544960eaa114d34e5e83ab19e180c8efcd299
  • Fix bug in SetOffsetVacuumLimit() triggered by find_multixact_start() failure. Previously, if find_multixact_start() failed, SetOffsetVacuumLimit() would install 0 into MultiXactState->offsetStopLimit if it previously succeeded. Luckily, there are no known cases where find_multixact_start() will return an error in 9.5 and above. But if it were to happen, for example due to filesystem permission issues, it'd be somewhat bad: GetNewMultiXactId() could continue allocating mxids even if close to a wraparound, or it could erroneously stop allocating mxids, even if no wraparound is looming. The wrong value would be corrected the next time SetOffsetVacuumLimit() is called, or by a restart. Reported-By: Noah Misch, although this is not his preferred fix Discussion: 20151210140450.GA22278@alap3.anarazel.de Backpatch: 9.5, where the bug was introduced as part of 4f627f http://git.postgresql.org/pg/commitdiff/cca705a5d93446e1a9c775b94c7d5900986c0488
  • Fix tab completion for ALTER ... TABLESPACE ... OWNED BY. Previously the completion used the wrong word to match 'BY'. This was introduced brokenly, in b2de2a. While at it, also add completion of IN TABLESPACE ... OWNED BY and fix comments referencing nonexistent syntax. Reported-By: Michael Paquier Author: Michael Paquier and Andres Freund Discussion: CAB7nPqSHDdSwsJqX0d2XzjqOHr==HdWiubCi4L=Zs7YFTUne8w@mail.gmail.com Backpatch: 9.4, like the commit introducing the bug http://git.postgresql.org/pg/commitdiff/130d94a7b868f5b6df512e5fde94a64e5e71178b

Heikki Linnakangas pushed:

  • Fix out-of-memory error handling in ParameterDescription message processing. If libpq ran out of memory while constructing the result set, it would hang, waiting for more data from the server, which might never arrive. To fix, distinguish between out-of-memory error and not-enough-data cases, and give a proper error message back to the client on OOM. There are still similar issues in handling COPY start messages, but let's handle that as a separate patch. Michael Paquier, Amit Kapila and me. Backpatch to all supported versions. http://git.postgresql.org/pg/commitdiff/7b96bf445a42b1cb2a435854f9825c38253f79a2

Kevin Grittner pushed:

  • Remove xmlparse(document '') test This one test was behaving differently between the ubuntu fix for CVE-2015-7499 and the base "expected" file. It's not worth having yet another version of the expected file for this test, so drop it. Perhaps at some point when all distros have settled down to the same behavior on this test, it can be restored. Problem found by me on libxml2 (2.9.1+dfsg1-3ubuntu4.6). Solution suggested by Tom Lane. Backpatch to 9.5, where the test was added. http://git.postgresql.org/pg/commitdiff/e2f1765ce0770e813971336bb4603099d24cbe57

Ãlvaro Herrera pushed:

  • Add missing CHECK_FOR_INTERRUPTS in lseg_inside_poly. Apparently, there are bugs in this code that cause it to loop endlessly. That bug still needs more research, but in the meantime it's clear that the loop is missing a check for interrupts so that it can be cancelled timely. Backpatch to 9.1 -- this has been missing since 49475aab8d0d. http://git.postgresql.org/pg/commitdiff/0d8f3d5d11f7304c82ce1383bbb491ec6abcffc4
  • Rework internals of changing a type's ownership. This is necessary so that REASSIGN OWNED does the right thing with composite types, to wit, that it also alters ownership of the type's pg_class entry -- previously, the pg_class entry remained owned by the original user, which caused later other failures such as the new owner's inability to use ALTER TYPE to rename an attribute of the affected composite. Also, if the original owner is later dropped, the pg_class entry becomes owned by a non-existant user which is bogus. To fix, create a new routine AlterTypeOwner_oid which knows whether to pass the request to ATExecChangeOwner or deal with it directly, and use that in shdepReassignOwner rather than calling AlterTypeOwnerInternal directly. AlterTypeOwnerInternal is now simpler in that it only modifies the pg_type entry and recurses to handle a possible array type; higher-level tasks are handled by either AlterTypeOwner directly or AlterTypeOwner_oid. I took the opportunity to add a few more objects to the test rig for REASSIGN OWNED, so that more cases are exercised. Additional ones could be added for superuser-only-ownable objects (such as FDWs and event triggers) but I didn't want to push my luck by adding a new superuser to the tests on a backpatchable bug fix. Per bug #13666 reported by Chris Pacejo. Backpatch to 9.5. (I would back-patch this all the way back, except that it doesn't apply cleanly in 9.4 and earlier because 59367fdf9 wasn't backpatched. If we decide that we need this in earlier branches too, we should backpatch both.) http://git.postgresql.org/pg/commitdiff/756e7b4c9db1fa713b886068643257c823baddaf

Tom Lane pushed:

  • Add missing cleanup logic in pg_rewind/t/005_same_timeline.pl test. Per Michael Paquier http://git.postgresql.org/pg/commitdiff/db81329eed6b1f54bbdd9049bcdba556f2b4737d
  • Update 9.5 release notes through today. Also do another round of copy-editing, and fix up remaining FIXME items. http://git.postgresql.org/pg/commitdiff/bfc7f5dd5dc641b475c27b872d6df21c20c75af1
  • Document use of Subject Alternative Names in SSL server certificates. Commit acd08d764 did not bother with updating the documentation. http://git.postgresql.org/pg/commitdiff/0625dbb0b96e2ecd557eb5bcdc458679123951db
  • Cope with Readline's failure to track SIGWINCH events outside of input. It emerges that libreadline doesn't notice terminal window size change events unless they occur while collecting input. This is easy to stumble over if you resize the window while using a pager to look at query output, but it can be demonstrated without any pager involvement. The symptom is that queries exceeding one line are misdisplayed during subsequent input cycles, because libreadline has the wrong idea of the screen dimensions. The safest, simplest way to fix this is to call rl_reset_screen_size() just before calling readline(). That causes an extra ioctl(TIOCGWINSZ) for every command; but since it only happens when reading from a tty, the performance impact should be negligible. A more valid objection is that this still leaves a tiny window during entry to readline() wherein delivery of SIGWINCH will be missed; but the practical consequences of that are probably negligible. In any case, there doesn't seem to be any good way to avoid the race, since readline exposes no functions that seem safe to call from a generic signal handler --- rl_reset_screen_size() certainly isn't. It turns out that we also need an explicit rl_initialize() call, else rl_reset_screen_size() dumps core when called before the first readline() call. rl_reset_screen_size() is not present in old versions of libreadline, so we need a configure test for that. (rl_initialize() is present at least back to readline 4.0, so we won't bother with a test for it.) We would need a configure test anyway since libedit's emulation of libreadline doesn't currently include such a function. Fortunately, libedit seems not to have any corresponding bug. Merlin Moncure, adjusted a bit by me http://git.postgresql.org/pg/commitdiff/2ec477dc8108339dcb6bb944fa93d19cafb6fff7
  • Fix improper initialization order for readline. Turns out we must set rl_basic_word_break_characters *before* we call rl_initialize() the first time, because it will quietly copy that value elsewhere --- but only on the first call. (Love these undocumented dependencies.) I broke this yesterday in commit 2ec477dc8108339d; like that commit, back-patch to all active branches. Per report from Pavel Stehule. http://git.postgresql.org/pg/commitdiff/aee7705be5b75d8e7873a32c4a0dd0afe1ae5928
  • Adjust behavior of single-user -j mode for better initdb error reporting. Previously, -j caused the entire input file to be read in and executed as a single command string. That's undesirable, not least because any error causes the entire file to be regurgitated as the "failing query". Some experimentation suggests a better rule: end the command string when we see a semicolon immediately followed by two newlines, ie, an empty line after a query. This serves nicely to break up the existing examples such as information_schema.sql and system_views.sql. A limitation is that it's no longer possible to write such a sequence within a string literal or multiline comment in a file meant to be read with -j; but there are no instances of such a problem within the data currently used by initdb. (If someone does make such a mistake in future, it'll be obvious because they'll get an unterminated-literal or unterminated-comment syntax error.) Other than that, there shouldn't be any negative consequences; you're not forced to end statements that way, it's just a better idea in most cases. In passing, remove src/include/tcop/tcopdebug.h, which is dead code because it's not included anywhere, and hasn't been for more than ten years. One of the debug-support symbols it purported to describe has been unreferenced for at least the same amount of time, and the other is removed by this commit on the grounds that it was useless: forcing -j mode all the time would have broken initdb. The lack of complaints about that, or about the missing inclusion, shows that no one has tried to use TCOP_DONTUSENEWLINE in many years. http://git.postgresql.org/pg/commitdiff/66d947b9d302f1fd6de3d156e6ec61f52e1dc2cb
  • Use just one standalone-backend session for initdb's post-bootstrap steps. Previously, each subroutine in initdb fired up its own standalone backend session. Over time we'd grown as many as fifteen of these sessions, and the cumulative startup and shutdown work for them was getting pretty noticeable. Combining things so that all these steps share a single backend session cuts a good 10% off the total runtime of initdb, more if you're not fsync'ing. The main stumbling block to doing this before was that some of the sessions were run with -j and some not. The improved definition of -j mode implemented by my previous commit makes it possible to fix that by running all the post-bootstrap steps with -j; we just have to use double instead of single newlines to end command strings. (This is only absolutely necessary around the VACUUM and CREATE DATABASE steps, since those can't be run in a transaction block. But it seems best to make them all use double newlines so that the commands remain separate for error-reporting purposes.) A minor disadvantage is that since initdb can't tell how much of its output the backend has executed, we can no longer have the per-step progress reporting initdb used to print. But things are fast enough nowadays that that's not really all that useful anyway. In passing, add more const decoration to some of the static arrays in initdb.c. http://git.postgresql.org/pg/commitdiff/c4a8812cf64b142685e39a69694c5276601f40e4
  • Remove unreferenced function declarations. datapagemap_create() and datapagemap_destroy() were declared extern, but they don't actually exist anywhere. Per YUriy Zhuravlev and Michael Paquier. http://git.postgresql.org/pg/commitdiff/3d0c50ffa0bdb683c28bfe0e79d23d87111da2aa
  • Adopt a more compact, less error-prone notation for tab completion code. Replace tests like else if (pg_strcasecmp(prev4_wd, "CREATE") == 0 && pg_strcasecmp(prev3_wd, "TRIGGER") == 0 && (pg_strcasecmp(prev_wd, "BEFORE") == 0 || pg_strcasecmp(prev_wd, "AFTER") == 0)) with new notation like this: else if (TailMatches4("CREATE", "TRIGGER", MatchAny, "BEFORE|AFTER")) In addition, provide some macros COMPLETE_WITH_LISTn() to reduce the amount of clutter needed to specify a small number of predetermined completion alternatives. This makes the code substantially more compact: tab-complete.c gets over a thousand lines shorter in this patch, despite the addition of a couple of hundred lines of infrastructure for the new notations. The new way of specifying match rules seems a whole lot more readable and less error-prone, too. There's a lot more that could be done now to make matching faster and more reliable; for example I suspect that most of the TailMatches() rules should now be Matches() rules. That would allow them to be skipped after a single integer comparison if there aren't the right number of words on the line, and it would reduce the risk of unintended matches. But for now, (mostly) refrain from reworking any match rules in favor of just converting what we've got into the new notation. Thomas Munro, reviewed by Michael Paquier, some adjustments by me http://git.postgresql.org/pg/commitdiff/d37b816dc9e8f976c8913296781e08cbd45c5af1
  • Add missing COSTS OFF to EXPLAIN commands in rowsecurity.sql. Commit e5e11c8cc added a bunch of EXPLAIN statements without COSTS OFF to the regression tests. This is contrary to project policy since it results in unnecessary platform dependencies in the output (it's just luck that we didn't get buildfarm failures from it). Per gripe from Mike Wilson. http://git.postgresql.org/pg/commitdiff/654218138b819df66c1b90d39a12ca6a75b9ff65
  • Teach psql's tab completion to consider the entire input string. Up to now, the tab completion logic has only examined the last few words of the current input line; "last few" being originally as few as four words, but lately up to nine words. Furthermore, it only looked at what libreadline considers the current line of input, which made it rather myopic if you split your command across lines. This was tolerable, sort of, so long as the match patterns were only designed to consider the last few words of input; but with the recent addition of HeadMatches() and Matches() matching rules, we really have to do better if we want those to behave sanely. Hence, change the code to break the entire line down into words, and to include any previous lines in the command buffer along with the active readline input buffer. This will be a little bit slower than the previous coding, but some measurements say that even a query of several thousand characters can be parsed in a hundred or so microseconds on modern machines; so it's really not going to be significant for interactive tab completion. To reduce the cost some, I arranged to avoid the per-word malloc calls that used to occur: all the words are now kept in one malloc'd buffer. http://git.postgresql.org/pg/commitdiff/d854118c8df8c413d069f7e88bb01b9e18e4c8ed
  • Remove silly completion for "DELETE FROM tabname ...". psql offered USING, WHERE, and SET in this context, but SET is not a valid possibility here. Seems to have been a thinko in commit f5ab0a14ea83eb6c which added DELETE's USING option. http://git.postgresql.org/pg/commitdiff/99ccb2309263183f0f3d838b79f3e07ad8cc6a63

Stephen Frost pushed:

  • Collect the global OR of hasRowSecurity flags for plancache We carry around information about if a given query has row security or not to allow the plancache to use that information to invalidate a planned query in the event that the environment changes. Previously, the flag of one of the subqueries was simply being copied into place to indicate if the query overall included RLS components. That's wrong as we need the global OR of all subqueries. Fix by changing the code to match how fireRIRules works, which is results in OR'ing all of the flags. Noted by Tom. Back-patch to 9.5 where RLS was introduced. http://git.postgresql.org/pg/commitdiff/e5e11c8cca7ae298895430102217fa6d77cfb2a3
  • Improve CREATE POLICY documentation Clarify that SELECT policies are now applied when SELECT rights are required for a given query, even if the query is an UPDATE or DELETE query. Pointed out by Noah. Additionally, note the risk regarding concurrently open transactions where a relation which controls access to the rows of another relation are updated and the rows of the primary relation are also being modified. Pointed out by Peter Geoghegan. Back-patch to 9.5. http://git.postgresql.org/pg/commitdiff/43cd468cf01007f39312af05c4c92ceb6de8afd8

Robert Haas pushed:

  • Provide a way to predefine LWLock tranche IDs. It's a bit cumbersome to use LWLockNewTrancheId(), because the returned value needs to be shared between backends so that each backend can call LWLockRegisterTranche() with the correct ID. So, for built-in tranches, use a hard-coded value instead. This is motivated by an upcoming patch adding further built-in tranches. Andres Freund and Robert Haas http://git.postgresql.org/pg/commitdiff/3fed417452b226d9bd85a3a54d7056b06eb14897
  • Move buffer I/O and content LWLocks out of the main tranche. Move the content lock directly into the BufferDesc, so that locking and pinning a buffer touches only one cache line rather than two. Adjust the definition of BufferDesc slightly so that this doesn't make the BufferDesc any larger than one cache line (at least on platforms where a spinlock is only 1 or 2 bytes). We can't fit the I/O locks into the BufferDesc and stay within one cache line, so move those to a completely separate tranche. This leaves a relatively limited number of LWLocks in the main tranche, so increase the padding of those remaining locks to a full cache line, rather than allowing adjacent locks to share a cache line, hopefully reducing false sharing. Performance testing shows that these changes make little difference on laptop-class machines, but help significantly on larger servers, especially those with more than 2 sockets. Andres Freund, originally based on an earlier patch by Simon Riggs. Review and cosmetic adjustments (including heavy rewriting of the comments) by me. http://git.postgresql.org/pg/commitdiff/6150a1b08a9fe7ead2b25240be46dddeae9d98e1
  • Teach mdnblocks() not to create zero-length files. It's entirely surprising that mdnblocks() has the side effect of creating new files on disk, so let's make it not do that. One consequence of the old behavior is that, if running on a damaged cluster that is missing a file, mdnblocks() can recreate the file and allow a subsequent _mdfd_getseg() for a higher segment to succeed. This happens because, while mdnblocks() stops when it finds a segment that is shorter than 1GB, _mdfd_getseg() has no such check, and thus the empty file created by mdnblocks() can allow it to continue its traversal and find higher-numbered segments which remain. It might be a good idea for _mdfd_getseg() to actually verify that each segment it finds is exactly 1GB before proceeding to the next one, but that would involve some additional system calls, so for now I'm just doing this much. Patch by me, per off-list analysis by Kevin Grittner and Rahila Syed. Review by Andres Freund. http://git.postgresql.org/pg/commitdiff/049469e7e7cfe0c69d30385952e2576b63230283
  • Mark CHECK constraints declared NOT VALID valid if created with table. FOREIGN KEY constraints have behaved this way for a long time, but for some reason the behavior of CHECK constraints has been inconsistent up until now. Amit Langote and Amul Sul, with assorted tweaks by me. http://git.postgresql.org/pg/commitdiff/f27a6b15e6566fba7748d0d9a3fc5bcfd52c4a1b
  • Speed up CREATE INDEX CONCURRENTLY's TID sort. Encode TIDs as 64-bit integers to speed up comparisons. This seems to speed things up on all platforms, but is even more beneficial when 8-byte integers are passed by value. Peter Geoghegan. Design suggestions and review by Tom Lane. Review also by Simon Riggs and by me. http://git.postgresql.org/pg/commitdiff/b648b70342fbe712383e8cd76dc8f7feaba9aaa3
  • Fix typo in comment. Amit Langote http://git.postgresql.org/pg/commitdiff/9a51698bae86f748279ecedcae018925b5af5b2d
  • Fix copy-and-paste error in logical decoding callback. This could result in the error context misidentifying where the error actually occurred. Craig Ringer http://git.postgresql.org/pg/commitdiff/4496226782c47e78b428babbcc16dec4f7329f2b
  • Fix TupleQueueReaderNext not to ignore its nowait argument. This was a silly goof on my (rhaas's) part. Report and fix by Rushabh Lathia. http://git.postgresql.org/pg/commitdiff/2bdfcb52c5d1446a1f19cc8bf16d44911658bcac
  • Remove duplicate word. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/6e7b335930200f71115fccd4903d04fe4de42021
  • pgbench: Change terminology from "threshold" to "parameter". Per a recommendation from Tomas Vondra, it's more helpful to refer to the value that determines how skewed a Gaussian or exponential distribution is as a parameter rather than a threshold. Since it's not quite too late to get this right in 9.5, where it was introduced, back-patch this. Most of the patch changes only comments and documentation, but a few pgbench messages are altered to match. Fabien Coelho, reviewed by Michael Paquier and by me. http://git.postgresql.org/pg/commitdiff/3c7042a7d7871b47dae3c9777c8020e41dedee89

Teodor Sigaev pushed:

Peter Eisentraut pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Amit Langote sent in another revision of a patch to fix an error in find_inheritance_children().

Craig Ringer sent in another revision of a patch to add logical decoding for sequence advances.

Tomas Vondra sent in a patch to postpone building buckets to the end of Hash (in HashJoin).

Aleksander Alekseev sent in another revision of a patch to speed up identification of resource owners, one outcome of which will be to speed up accesses to tables with many partitions.

Daniel Verité and Pavel Stěhule traded patches to implement \crosstabview in psql.

Michael Paquier sent in three more revisions of a patch to extend pgbench expressions with functions.

Andreas Karlsson sent in a patch to make it possible to cancel a query which is running the crypt() function with the bf or xdes hashing algorithm by adding a CHECK_FOR_INTERRUPTS() call every round.

Andres Freund sent in a patch to make a faster PageIsVerified() for the all zeroes case and improve scalability of md.c for large relations.

David Rowley sent in three more revisions of a patch to improve for joins where outer side is unique.

Aleksander Alekseev sent in a patch to ensure that the initial sizes of the PROCLOCK and LOCK hash tables be the same as max_size, saving time that would have been spent expanding them.

Haribabu Kommi sent in another revision of a patch to add parallel aggregation.

Tomas Vondra sent in a patch to add a bloom filter in Hash Joins with batches.

Haribabu Kommi sent in another revision of a patch to add a pg_hba_lookup() function to get all matching pg_hba.conf entries.

Fabien COELHO sent in another revision of a patch to add pgbench stats per script, etc.

Haribabu Kommi sent in another revision of a patch to implement multi-tenancy with RLS.

Jesper Pedersen sent in another revision of a patch to add new LWLOCK_STATS statistics.

Tatsuro Yamada sent in a patch to fix a typo in regress/sql/privileges.sql.

David Rowley sent in another revision of a patch to implement combining aggregates, useful in parallelizing aggregation.

SAWADA Masahiko sent in another revision of a patch to support for N synchronous standby servers for N>1.

Amit Langote sent in a patch to fix a comment typo in pg_upgrade.c.

Aleksander Alekseev sent in a patch to fix lock contention for HASHHDR.mutex.

Michael Paquier sent in a patch to fix a psql call in pg_upgrade.

SAWADA Masahiko sent in another revision of a patch to add a "frozen" bit to the visibility map.

Peter Geoghegan sent in another revision of a patch to reuse abbreviated keys during second pass of ordered [set] aggregates.

Ashutosh Bapat and Robert Haas traded patches to allow foreign scans to be sorted on the remote side.

Alexander Korotkov sent in a patch to improve statistics for array types.

Michael Paquier sent in a patch to add a system view and function to report WAL receiver.

Robert Haas sent in another revision of a patch to implement parallel joins, and better parallel explain.

Artur Zakirov sent in a patch to add fuzzy substring searching with the pg_trgm extension.

David Rowley sent in a patch to wrap error code paths in unlikely().

Mithun Cy sent in a patch to cache data in GetSnapshotData().

Pavel Stěhule sent in another revision of a patch to add a pg_size_bytes() to transform human-readable byte sizes into bytes.

Michael Paquier sent in another revision of a patch to reduce some unneeded WAL logging in hot standby.

par N Bougain le lundi 21 décembre 2015 à 15h04

jeudi 17 décembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 13 décembre 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Détails et appel à conférenciers ci-après : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • La première conférence PostgreSQL pan-asiatique se tiendra en mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/
  • La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/
  • La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers a été lancé : http://www.pgcon.org/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151214071213.GB22851@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Ãlvaro Herrera pushed:

  • Clean up some problems in new Perl test code. Noted by Tom Lane: - PostgresNode had a BEGIN block which created files, contrary to perlmod suggestions to do that only on INIT blocks. - Assign ports randomly rather than starting from 90600. Noted by Noah Misch: - Change use of no-longer-set PGPORT environment variable to $node->port - Don't start a server in pg_controldata test - PostgresNode was reading the PID file incorrectly; test the right thing, and chomp the line we read from the PID file. - Remove an unused $devnull variable - Use 'pg_ctl kill' instead of "kill" directly, for Windos portability. - Make server log names more informative. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/9821492ee417a5910a60f3d1f2ed24c062eab4e0
  • PostgresNode: wrap correctly around port number range end. Per note from Tom Lane http://git.postgresql.org/pg/commitdiff/7ac5d9b31637b1856c7ac9cb625bcca694a2790a
  • Fix commit timestamp initialization. This module needs explicit initialization in order to replay WAL records in recovery, but we had broken this recently following changes to make other (stranger) scenarios work correctly. To fix, rework the initialization sequence so that it always takes place before WAL replay commences for both master and standby. I could have gone for a more localized fix that just added a "startup" call for the master server, but it seemed better to restructure the existing callers as well so that the whole thing made more sense. As a drawback, there is more control logic in xlog.c now than previously, but doing otherwise meant passing down the ControlFile flag, which seemed uglier as a whole. This also meant adding a check to not re-execute ActivateCommitTs if it had already been called. Reported by Fujii Masao. Backpatch to 9.5. http://git.postgresql.org/pg/commitdiff/69e7235c93e2965cc0e17186bd044e4c54997c19
  • For REASSIGN OWNED for foreign user mappings As reported in bug #13809 by Alexander Ashurkov, the code for REASSIGN OWNED hadn't gotten word about user mappings. Deal with them in the same way default ACLs do, which is to ignore them altogether; they are handled just fine by DROP OWNED. The other foreign object cases are already handled correctly by both commands. Also add a REASSIGN OWNED statement to foreign_data test to exercise the foreign data objects. (The changes are just before the "cleanup" phase, so it shouldn't remove any existing live test.) Reported by Alexander Ashurkov, then independently by Jaime Casanova. http://git.postgresql.org/pg/commitdiff/8c1615531f4945e5af78ddf1d43af11b6d7b48fd

Tom Lane pushed:

  • Fix another oversight in checking if a join with LATERAL refs is legal. It was possible for the planner to decide to join a LATERAL subquery to the outer side of an outer join before the outer join itself is completed. Normally that's fine because of the associativity rules, but it doesn't work if the subquery contains a lateral reference to the inner side of the outer join. In such a situation the outer join *must* be done first. join_is_legal() missed this consideration and would allow the join to be attempted, but the actual path-building code correctly decided that no valid join path could be made, sometimes leading to planner errors such as "failed to build any N-way joins". Per report from Andreas Seltenreich. Back-patch to 9.3 where LATERAL support was added. http://git.postgresql.org/pg/commitdiff/7e19db0c09719d7919a8fdd96a1fffe7efd2df93
  • Simplify LATERAL-related calculations within add_paths_to_joinrel(). While convincing myself that commit 7e19db0c09719d79 would solve both of the problems recently reported by Andreas Seltenreich, I realized that add_paths_to_joinrel's handling of LATERAL restrictions could be made noticeably simpler and faster if we were to retain the minimum possible parameterization for each joinrel (that is, the set of relids supplying unsatisfied lateral references in it). We already retain that for baserels, in RelOptInfo.lateral_relids, so we can use that field for joinrels too. I re-pgindent'd the files touched here, which affects some unrelated comments. This is, I believe, just a minor optimization not a bug fix, so no back-patch. http://git.postgresql.org/pg/commitdiff/edca44b1525b3d591263d032dc4fe500ea771e0e
  • Avoid odd portability problem in TestLib.pm's slurp_file function. For unclear reasons, this function doesn't always read the expected data in some old Perl versions. Rewriting it to avoid use of ARGV seems to dodge the problem, and this version is clearer anyway if you ask me. In passing, also improve error message in adjacent append_to_file function. http://git.postgresql.org/pg/commitdiff/938d797b84467ebbaed432da72d1d7f7bb266110
  • Make failure to open psql's --log-file fatal. Commit 344cdff2c made failure to open the target of --output fatal. For consistency, the --log-file switch should behave similarly. Like the previous commit, back-patch to 9.5 but no further. Daniel Verite http://git.postgresql.org/pg/commitdiff/521f0458dcd7bb28636250cdfce9b1a3020cff4c
  • Accept flex > 2.5.x on Windows, too. Commit 32f15d05c fixed this in configure, but missed the similar check in the MSVC scripts. Michael Paquier, per report from Victor Wagner http://git.postgresql.org/pg/commitdiff/9c779c49e39372c21214ad2dc5864dd78caee82d
  • Still more fixes for planner's handling of LATERAL references. More fuzz testing by Andreas Seltenreich exposed that the planner did not cope well with chains of lateral references. If relation X references Y laterally, and Y references Z laterally, then we will have to scan X on the inside of a nestloop with Z, so for all intents and purposes X is laterally dependent on Z too. The planner did not understand this and would generate intermediate joins that could not be used. While that was usually harmless except for wasting some planning cycles, under the right circumstances it would lead to "failed to build any N-way joins" or "could not devise a query plan" planner failures. To fix that, convert the existing per-relation lateral_relids and lateral_referencers relid sets into their transitive closures; that is, they now show all relations on which a rel is directly or indirectly laterally dependent. This not only fixes the chained-reference problem but allows some of the relevant tests to be made substantially simpler and faster, since they can be reduced to simple bitmap manipulations instead of searches of the LateralJoinInfo list. Also, when a PlaceHolderVar that is due to be evaluated at a join contains lateral references, we should treat those references as indirect lateral dependencies of each of the join's base relations. This prevents us from trying to join any individual base relations to the lateral reference source before the join is formed, which again cannot work. Andreas' testing also exposed another oversight in the "dangerous PlaceHolderVar" test added in commit 85e5e222b1dd02f1. Simply rejecting unsafe join paths in joinpath.c is insufficient, because in some cases we will end up rejecting *all* possible paths for a particular join, again leading to "could not devise a query plan" failures. The restriction has to be known also to join_is_legal and its cohort functions, so that they will not select a join for which that will happen. I chose to move the supporting logic into joinrels.c where the latter functions are. Back-patch to 9.3 where LATERAL support was introduced. http://git.postgresql.org/pg/commitdiff/acfcd45cacb6df23edba4cb3753a2be594238a99
  • Get rid of the planner's LateralJoinInfo data structure. I originally modeled this data structure on SpecialJoinInfo, but after commit acfcd45cacb6df23 that looks like a pretty poor decision. All we really need is relid sets identifying laterally-referenced rels; and most of the time, what we want to know about includes indirect lateral references, a case the LateralJoinInfo data was unsuited to compute with any efficiency. The previous commit redefined RelOptInfo.lateral_relids as the transitive closure of lateral references, so that it easily supports checking indirect references. For the places where we really do want just direct references, add a new RelOptInfo field direct_lateral_relids, which is easily set up as a copy of lateral_relids before we perform the transitive closure calculation. Then we can just drop lateral_info_list and LateralJoinInfo and the supporting code. This makes the planner's handling of lateral references noticeably more efficient, and shorter too. Such a change can't be back-patched into stable branches for fear of breaking extensions that might be looking at the planner's data structures; but it seems not too late to push it into 9.5, so I've done so. http://git.postgresql.org/pg/commitdiff/4fcf48450d38e28e69e629c779a7866183d8ea41
  • Install our "missing" script where PGXS builds can find it. This allows sane behavior in a PGXS build done on a machine where build tools such as bison are missing. Jim Nasby http://git.postgresql.org/pg/commitdiff/dccf8e9e608824ce15d3b2d7ee63afbf1147c5e9
  • Add an expected-file to match behavior of latest libxml2. Recent releases of libxml2 do not provide error context reports for errors detected at the very end of the input string. This appears to be a bug, or at least an infelicity, introduced by the fix for libxml2's CVE-2015-7499. We can hope that this behavioral change will get undone before too long; but the security patch is likely to spread a lot faster/further than any follow-on cleanup, which means this behavior is likely to be present in the wild for some time to come. As a stopgap, add a variant regression test expected-file that matches what you get with a libxml2 that acts this way. http://git.postgresql.org/pg/commitdiff/085423e3e326da1b52f41aa86126f2a064a7db25
  • Doc: update external URLs for PostGIS project. Paul Ramsey http://git.postgresql.org/pg/commitdiff/6d96cd077bedcf822db7348bc564b44df8b2ab23
  • Code and docs review for multiple -c and -f options in psql. Commit d5563d7df94488bf drew complaints from Coverity, which quite correctly complained that one copy of each -c or -f string was being leaked. What's more, simple_action_list_append was allocating enough space for still a third copy of each string as part of the SimpleActionListCell, even though that coding method had been superseded by a separate strdup operation. There were some other minor coding infelicities too. The documentation needed more work as well, eg it forgot to explain that -c causes psql not to accept any interactive input. http://git.postgresql.org/pg/commitdiff/fcbbf82d2b6caf7b156f2ec35b322e23caf1e99e
  • Docs: document that psql's "\i -" means read from stdin. This has worked that way for a long time, maybe always, but you would not have known it from the documentation. Also back-patch the notes I added to HEAD earlier today about behavior of the "-f -" switch, which likewise have been valid for many releases. http://git.postgresql.org/pg/commitdiff/7bd149ce2aa6af8910604eb45cc202798f563781

Robert Haas pushed:

  • Allow foreign and custom joins to handle EvalPlanQual rechecks. Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 provided basic infrastructure for allowing a foreign data wrapper or custom scan provider to replace a join of one or more tables with a scan. However, this infrastructure failed to take into account the need for possible EvalPlanQual rechecks, and ExecScanFetch would fail an assertion (or just overwrite memory) if such a check was attempted for a plan containing a pushed-down join. To fix, adjust the EPQ machinery to skip some processing steps when scanrelid == 0, making those the responsibility of scan's recheck method, which also has the responsibility in this case of correctly populating the relevant slot. To allow foreign scans to gain control in the right place to make use of this new facility, add a new, optional RecheckForeignScan method. Also, allow a foreign scan to have a child plan, which can be used to correctly populate the slot (or perhaps for something else, but this is the only use currently envisioned). KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and Kyotaro Horiguchi. http://git.postgresql.org/pg/commitdiff/385f337c9f39b21dca96ca4770552a10a6d5af24
  • psql: Support multiple -c and -f options, and allow mixing them. To support this, we must reconcile some historical anomalies in the behavior of -c. In particular, as a backward-incompatibility, -c no longer implies --no-psqlrc. Pavel Stehule (code) and Catalin Iacob (documentation). Review by Michael Paquier and myself. Proposed behavior per Tom Lane. http://git.postgresql.org/pg/commitdiff/d5563d7df94488bf0ab52ac0678e8a07e5b8297e
  • Allow EXPLAIN (ANALYZE, VERBOSE) to display per-worker statistics. The original parallel sequential scan commit included only very limited changes to the EXPLAIN output. Aggregated totals from all workers were displayed, but there was no way to see what each individual worker did or to distinguish the effort made by the workers from the effort made by the leader. Per a gripe by Thom Brown (and maybe others). Patch by me, reviewed by Amit Kapila. http://git.postgresql.org/pg/commitdiff/b287df70e4080350aa471ecca428be145581dd4d
  • Remove redundant sentence. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/c00239ea6a0612cd7d1cd604ffc5b6f73ef9fb09
  • Fix typo. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/348bcd864530b4860d0fd0ffe8532f590a5779f7
  • Improve ALTER POLICY tab completion. Complete "ALTER POLICY" with a policy name, as we do for DROP POLICY. And, complete "ALTER POLICY polname ON" with a table name that has such a policy, as we do for DROP POLICY, rather than with any table name at all. Masahiko Sawada http://git.postgresql.org/pg/commitdiff/8b469bd7c4dc3649c74b746545210b435df319cb

Kevin Grittner pushed:

  • Improve performance in freeing memory contexts The single linked list of memory contexts could result in O(N^2) performance to free a set of contexts if they were not freed in reverse order of creation. In many cases the reverse order was used, but there were some significant exceptions that caused real- world performance problems. Rather than requiring all callers to care about the order in which contexts were freed, and hunting down and changing all existing cases where the wrong order was used, we add one pointer per memory context so that the implementation details are not so visible. Jan Wieck http://git.postgresql.org/pg/commitdiff/25c539233044c235e97fd7c9dc600fb5f08fe065

Andres Freund pushed:

  • Fix ON CONFLICT UPDATE bug breaking AFTER UPDATE triggers. ExecOnConflictUpdate() passed t_ctid of the to-be-updated tuple to ExecUpdate(). That's problematic primarily because of two reason: First and foremost t_ctid could point to a different tuple. Secondly, and that's what triggered the complaint by Stanislav, t_ctid is changed by heap_update() to point to the new tuple version. The behavior of AFTER UPDATE triggers was therefore broken, with NEW.* and OLD.* tuples spuriously identical within AFTER UPDATE triggers. To fix both issues, pass a pointer to t_self of a on-stack HeapTuple instead. Fixing this bug lead to one change in regression tests, which previously failed due to the first issue mentioned above. There's a reasonable expectation that test fails, as it updates one row repeatedly within one INSERT ... ON CONFLICT statement. That is only possible if the second update is triggered via ON CONFLICT ... SET, ON CONFLICT ... WHERE, or by a WITH CHECK expression, as those are executed after ExecOnConflictUpdate() does a visibility check. That could easily be prohibited, but given it's allowed for plain UPDATEs and a rare corner case, it doesn't seem worthwhile. Reported-By: Stanislav Grozev Author: Andres Freund and Peter Geoghegan Discussion: CAA78GVqy1+LisN-8DygekD_Ldfy=BJLarSpjGhytOsgkpMavfQ@mail.gmail.com Backpatch: 9.5, where ON CONFLICT was introduced http://git.postgresql.org/pg/commitdiff/84ac126ee728ede5b6370d60dd2b1c299f49ed2f
  • Fix ALTER TABLE ... SET TABLESPACE for unlogged relations. Changing the tablespace of an unlogged relation did not WAL log the creation and content of the init fork. Thus, after a standby is promoted, unlogged relation cannot be accessed anymore, with errors like: ERROR: 58P01: could not open file "pg_tblspc/...": No such file or directory Additionally the init fork was not synced to disk, independent of the configured wal_level, a relatively small durability risk. Investigation of that problem also brought to light that, even for permanent relations, the creation of !main forks was not WAL logged, i.e. no XLOG_SMGR_CREATE record were emitted. That mostly turns out not to be a problem, because these files were created when the actual relation data is copied; nonexistent files are not treated as an error condition during replay. But that doesn't work for empty files, and generally feels a bit haphazard. Luckily, outside init and main forks, empty forks don't occur often or are not a problem. Add the required WAL logging and syncing to disk. Reported-By: Michael Paquier Author: Michael Paquier and Andres Freund Discussion: 20151210163230.GA11331@alap3.anarazel.de Backpatch: 9.1, where unlogged relations were introduced http://git.postgresql.org/pg/commitdiff/f54d0629ec8bc17de932c83fc9872c31bb8da68f
  • Fix bug leading to restoring unlogged relations from empty files. At the end of crash recovery, unlogged relations are reset to the empty state, using their init fork as the template. The init fork is copied to the main fork without going through shared buffers. Unfortunately WAL replay so far has not necessarily flushed writes from shared buffers to disk at that point. In normal crash recovery, and before the introduction of 'fast promotions' in fd4ced523 / 9.3, the END_OF_RECOVERY checkpoint flushes the buffers out in time. But with fast promotions that's not the case anymore. To fix, force WAL writes targeting the init fork to be flushed immediately (using the new FlushOneBuffer() function). In 9.5+ that flush can centrally be triggered from the code dealing with restoring full page writes (XLogReadBufferForRedoExtended), in earlier releases that responsibility is in the hands of XLOG_HEAP_NEWPAGE's replay function. Backpatch to 9.1, even if this currently is only known to trigger in 9.3+. Flushing earlier is more robust, and it is advantageous to keep the branches similar. Typical symptoms of this bug are errors like 'ERROR: index "..." contains unexpected zero page at block 0' shortly after promoting a node. Reported-By: Thom Brown Author: Andres Freund and Michael Paquier Discussion: 20150326175024.GJ451@alap3.anarazel.de Backpatch: 9.1- http://git.postgresql.org/pg/commitdiff/e3f4cfc7aa33f40b2b283676c9f0d5bf7ef08ccd

Peter Eisentraut pushed:

Stephen Frost pushed:

  • Handle dependencies properly in ALTER POLICY. ALTER POLICY hadn't fully considered partial policy alternation (eg: change just the roles on the policy, or just change one of the expressions) when rebuilding the dependencies. Instead, it would happily remove all dependencies which existed for the policy and then only recreate the dependencies for the objects referred to in the specific ALTER POLICY command. Correct that by extracting and building the dependencies for all objects referenced by the policy, regardless of if they were provided as part of the ALTER POLICY command or were already in place as part of the pre-existing policy. http://git.postgresql.org/pg/commitdiff/ed8bec915ec570bd90d86710392fe0c181fd08fe
  • Handle policies during DROP OWNED BY. DROP OWNED BY handled GRANT-based ACLs but was not removing roles from policies. Fix that by having DROP OWNED BY remove the role specified from the list of roles the policy (or policies) apply to, or the entire policy (or policies) if it only applied to the role specified. As with ACLs, the DROP OWNED BY caller must have permission to modify the policy or a WARNING is thrown and no change is made to the policy. http://git.postgresql.org/pg/commitdiff/833728d4c8832f1d37e7aeaa723c8bc4045df32e

Magnus Hagander pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Michael Paquier sent in four more revisions of a patch to add in-core regression tests for replication, cascading, archiving, PITR, etc.

Stas Kelvich sent in another revision of a patch to add tsvector editing functions.

Stas Kelvich sent in another revision of a patch to add KNN support to the cube extension.

Fujii Masao sent in another revision of a patch to fix some error messages when connecting to older DBs.

YUriy Zhuravlev sent in another revision of a patch to extend the array slice syntax.

Haribabu Kommi sent in two more revisions of a patch to add a pg_hba_lookup function to get all matching pg_hba.conf entries.

Amit Langote and Amul Sul traded patches to fix some issues around NOT VALID constraints.

Etsuro Fujita sent in a patch to adjust a comment in src/backend/optimizer/plan/setrefs.c to be more inclusive.

SAWADA Masahiko sent in a patch to add tab completion to psql for the new RLS feature.

Pavel Stěhule sent in three more revisions of a patch to add a \crosstabview command to psql.

Alexander Korotkov sent in another revision of a patch to move PinBuffer and UnpinBuffer to atomics.

Gregory Stark sent in a patch to add logical tape size to TRACE_SORT output and replace the selection sort base case with the optimal open case statement.

Victor Wagner sent in another revision of a patch to implement failover on the libpq connect level.

Stas Kelvich sent in two revisions of a patch to speed up 2PC transactions.

Robert Haas sent in a patch to fix an issue where a wrongly-deleted file could cause index scans and sequential scans to have totally different ideas about how many accessible blocks there are in a relation.

Alexander Korotkov sent in another revision of a patch to help make ResourceOwner more efficient for tables with many (thousands) partitions.

SAWADA Masahiko sent in two more revisions of a patch to support multiple synchronous standby servers.

Alexander Korotkov sent in two more revisions of a patch to rework the access method interface.

Kyotaro HORIGUCHI sent in another revision of a patch to make tab-complete easier to maintain using regex.

Thomas Munro sent in two more revisions of a patch to make tab-complete easier to maintain using macros.

Haribabu Kommi sent in two more revisions of a patch to implement parallel aggregates.

Aleksander Alekseev sent in a patch to fix an issue where there was unnecessary lock contention for HASHHDR.mutex.

Caleb Welton sent in a patch to make bootstrap (bki) changes less painful.

Michael Paquier sent in a patch to fix a bug in psql's tab completion for ALL IN TABLESPACE.

Amit Kapila sent in another revision of a patch to speed up clog access by increasing CLOG buffers.

Andres Freund sent in another revision of a patch to rework the way multixact truncations work.

Tomas Vondra sent in a patch to track the last known XLOG segment in the control file.

Peter Geoghegan sent in a patch to speed up CREATE INDEX CONCURRENTLY's TID sort.

Andres Freund sent in a patch to define an enum of builtin LWLock tranches, and move buffer LWLocks out of the main array and increase padding of locks.

Tom Lane refactored the way bootstrapping is done for more sanity.

Tomas Vondra sent in a patch to add a pg_current_xlog_flush_location() function.

Michael Paquier sent in a patch to fix some bugs in psql's tab completion for CREATE INDEX.

Craig Ringer sent in a patch to add logical decoding for sequence advances.

par N Bougain le jeudi 17 décembre 2015 à 00h46

mercredi 9 décembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 6 décembre 2015

La PGConf US 2016 aura lieu les 18, 19 et 20 avril à New-York. L'appel à conférenciers expire au 31 janvier 2016, 23:59EST : http://www.pgconf.us/2016/

La PGCon 2016 se tiendra du 17 au 21 mai 2016 à Ottawa. L'appel à conférenciers sera lancé mardi : http://www.pgcon.org/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. https://www.socallinuxexpo.org/scale/14x/cfp
  • FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Détails et appel à conférenciers ci-après : http://fosdem2016.pgconf.eu/
  • Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/
  • La première conférence PostgreSQL pan-asiatique se tiendra en mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151207050056.GA6554@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Avoid caching expression state trees for domain constraints across queries. In commit 8abb3cda0ddc00a0ab98977a1633a95b97068d4e I attempted to cache the expression state trees constructed for domain CHECK constraints for the life of the backend (assuming the domain's constraints don't get redefined). However, this turns out not to work very well, because execQual.c will run those state trees with ecxt_per_query_memory pointing to a query-lifespan context, and in some situations we'll end up with pointers into that context getting stored into the state trees. This happens in particular with SQL-language functions, as reported by Emre Hasegeli, but there are many other cases. To fix, keep only the expression plan trees for domain CHECK constraints in the typcache's data structure, and revert to performing ExecInitExpr (at least) once per query to set up expression state trees in the query's context. Eventually it'd be nice to undo this, but that will require some careful thought about memory management for expression state trees, and it seems far too late for any such redesign in 9.5. This way is still much more efficient than what happened before 8abb3cda0. http://git.postgresql.org/pg/commitdiff/ec7eef6b1103f92aea94aa79aabd8e0e87973010
  • Rework wrap-width calculation in psql's print_aligned_vertical() function. This area was rather heavily whacked around in 6513633b9 and follow-on commits, and it was showing it, because the logic to calculate the allowable data width in wrapped expanded mode had only the vaguest relationship to the logic that was actually printing the data. It was not very close to being right about the conditions requiring overhead columns to be added. Aside from being wrong, it was pretty unreadable and under-commented. Rewrite it so it corresponds to what the printing code actually does. In passing, remove a couple of dead tests in the printing logic, too. Per a complaint from Jeff Janes, though this doesn't look much like his patch because it fixes a number of other corner-case bogosities too. One such fix that's visible in the regression test results is that although the code was attempting to enforce a minimum data width of 3 columns, it sometimes left less space than that available. http://git.postgresql.org/pg/commitdiff/0e0776bc99553ff229e0d536ed8c78ab9db62464
  • Further adjustment to psql's print_aligned_vertical() function. We should ignore output_columns unless it's greater than zero. A zero means we couldn't get any information from ioctl(TIOCGWINSZ); in that case the expected behavior is to print the data at native width, not to wrap it at the smallest possible value. print_aligned_text() gets this consideration right, but print_aligned_vertical() lost track of this detail somewhere along the line. http://git.postgresql.org/pg/commitdiff/2287b874546931e2f33cddcafff2bde58459da1b
  • Use "g" not "f" format in ecpg's PGTYPESnumeric_from_double(). The previous coding could overrun the provided buffer size for a very large input, or lose precision for a very small input. Adopt the methodology that's been in use in the equivalent backend code for a long time. Per private report from Bas van Schaik. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/db4a5cfc76206db82d0b929d96c53de229ef1aa4
  • Further tweaking of print_aligned_vertical(). Don't force the data width to extend all the way to the right margin if it doesn't need to. This reverts the behavior in non-wrapping cases to be what it was in 9.4. Also, make the logic that ensures the data line width is at least equal to the record-header line width a little less obscure. In passing, avoid possible calculation of log10(0). Probably that's harmless, given the lack of field complaints, but it seems risky: conversion of NaN to an integer isn't well defined. http://git.postgresql.org/pg/commitdiff/95708e1d8e38bc1f7f6ee02c38dd86dd53b9663c
  • Make gincostestimate() cope with hypothetical GIN indexes. We tried to fetch statistics data from the index metapage, which does not work if the index isn't actually present. If the index is hypothetical, instead extrapolate some plausible internal statistics based on the index page count provided by the index-advisor plugin. There was already some code in gincostestimate() to invent internal stats in this way, but since it was only meant as a stopgap for pre-9.1 GIN indexes that hadn't been vacuumed since upgrading, it was pretty crude. If we want it to support index advisors, we should try a little harder. A small amount of testing says that it's better to estimate the entry pages as 90% of the index, not 100%. Also, estimating the number of entries (keys) as equal to the heap tuple count could be wildly wrong in either direction. Instead, let's estimate 100 entries per entry page. Perhaps someday somebody will want the index advisor to be able to provide these numbers more directly, but for the moment this should serve. Problem report and initial patch by Julien Rouhaud; modified by me to invent less-bogus internal statistics. Back-patch to all supported branches, since we've supported index advisors since 9.0. http://git.postgresql.org/pg/commitdiff/7fb008c5ee59b04005a8ee6c1006249f729e2b03
  • Fix behavior of printTable() and friends with externally-invoked pager. The formatting modes that depend on knowledge of the terminal window width did not work right when printing a query result that's been fetched in sections (as a result of FETCH_SIZE). ExecQueryUsingCursor() would force use of the pager as soon as there's more than one result section, and then print.c would see an output file pointer that's not stdout and incorrectly conclude that the terminal window width isn't relevant. This has been broken all along for non-expanded "wrapped" output format, and as of 9.5 the issue affects expanded mode as well. The problem also caused "\pset expanded auto" mode to invariably *not* switch to expanded output in a segmented result, which seems to me to be exactly backwards. To fix, we need to pass down an "is_pager" flag to inform the print.c subroutines that some calling level has already replaced stdout with a pager pipe, so they should (a) not do that again and (b) nonetheless honor the window size. (Notably, this makes the first is_pager test in print_aligned_text() not be dead code anymore.) This patch is a bit invasive because there are so many existing calls of printQuery()/printTable(), but fortunately all but a couple can just pass "false" for the added parameter. Back-patch to 9.5 but no further. Given the lack of field complaints, it's not clear that we should change the behavior in stable branches. Also, the API change for printQuery()/printTable() might possibly break third-party code, again something we don't like to do in stable branches. However, it's not quite too late to do this in 9.5, and with the larger scope of the problem there, it seems worth doing. http://git.postgresql.org/pg/commitdiff/d8ff060ecd5fc2dd20021743518d376a11a0bfd8
  • Clean up some psql issues around handling of the query output file. Formerly, if "psql -o foo" failed to open the output file "foo", it would print an error message but then carry on as though -o had not been specified at all. This seems contrary to expectation: a program that cannot open its output file normally fails altogether. Make psql do exit(1) after reporting the error. If "\o foo" failed to open "foo", it would print an error message but then reset the output file to stdout, as if the argument had been omitted. This is likewise pretty surprising behavior. Make it keep the previous output state, instead. psql keeps SIGPIPE interrupts disabled when it is writing to a pipe, either a pipe specified by -o/\o or a transient pipe opened for purposes such as using a pager on query output. The logic for this was too simple and could sometimes re-enable SIGPIPE when a -o pipe was still active, thus possibly leading to an unexpected psql crash later. Fixing the last point required getting rid of the kluge in PrintQueryTuples and ExecQueryUsingCursor whereby they'd transiently change the global queryFout state, but that seems like good cleanup anyway. Back-patch to 9.5 but not further; these are minor-enough issues that changing the behavior in stable branches doesn't seem appropriate. http://git.postgresql.org/pg/commitdiff/344cdff2c1541e7a1249299a33723aabeafa0b0c
  • Further improve documentation of the role-dropping process. In commit 1ea0c73c2 I added a section to user-manag.sgml about how to drop roles that own objects; but as pointed out by Stephen Frost, I neglected that shared objects (databases or tablespaces) may need special treatment. Fix that. Back-patch to supported versions, like the previous patch. http://git.postgresql.org/pg/commitdiff/63acfb79ab53d3833ad035728fb67a6c3018778f
  • Create TestLib.pm's tempdir underneath tmp_check/, not out in the open. This way, existing .gitignore entries and makefile clean actions will automatically apply to the tempdir, should it survive a TAP test run (which can happen if the user control-C's out of the run, for example). Michael Paquier, per a complaint from me http://git.postgresql.org/pg/commitdiff/db0723631ef1460e9e795c6d13abb19da403a3f1
  • Update xindex.sgml for recent additions to GIST opclass API. Commit d04c8ed9044ec added another support function to the GIST API, but overlooked mentioning it in xindex.sgml's summary of index support functions. Anastasia Lubennikova http://git.postgresql.org/pg/commitdiff/b0cfb02cec9650492365c6a9eca2dfce3671096d

Robert Haas pushed:

Teodor Sigaev pushed:

  • Use pg_rewind when target timeline was switched Allow pg_rewind to work when target timeline was switched. Now user can return promoted standby to old master. Target timeline history becomes a global variable. Index in target timeline history is used in function interfaces instead of specifying TLI directly. Thus, SimpleXLogPageRead() can easily start reading XLOGs from next timeline when current timeline ends. Author: Alexander Korotkov Review: Michael Paquier http://git.postgresql.org/pg/commitdiff/e50cda78404d6400b1326a996a4fabb144871151

Ãlvaro Herrera pushed:

  • Fix broken subroutine call in TestLib. Michael Paquier http://git.postgresql.org/pg/commitdiff/a2983cfd9d64d352c68431010b2525daf2b8bc79
  • Refactor Perl test code. The original code was a bit clunky; make it more amenable for further reuse by creating a new Perl package PostgresNode, which is an object-oriented representation of a single server, with some support routines such as init, start, stop, psql. This serves as a better basis on which to build further test code, and enables writing tests that use more than one server without too much complication. This commit modifies a lot of the existing test files, mostly to remove explicit calls to system commands (pg_ctl) replacing them with method calls of a PostgresNode object. The result is quite a bit more straightforward. Also move some initialization code to BEGIN and INIT blocks instead of having it straight in as top-level code. This commit also introduces package RecursiveCopy so that we can copy whole directories without having to depend on packages that may not be present on vanilla Perl 5.8 installations. I also ran perltidy on the modified files, which changes some code sites that are not otherwise touched by this patch. I tried to avoid this, but it ended up being more trouble than it's worth. Authors: Michael Paquier, Ãlvaro Herrera Review: Noah Misch http://git.postgresql.org/pg/commitdiff/1caef31d9e550408d0cbc5788a422dcb69736df5
  • Further tweak commit_timestamp behavior. As pointed out by Fujii Masao, we weren't quite there on a standby behaving sanely: first because we were failing to acquire the correct state in the case where no XLOG_PARAMETER_CHANGE message was sent (because a checkpoint had already happened after the setting was changed in the master, and then the standby was restarted); and second because promoting the standby with the feature enabled failed to activate it if the master had the feature disabled. This patch fixes both those misbehaviors hopefully without re-introducing any old problems. Also change the hint emitted in a standby together with the error message about the feature being disabled, to make it point out that the place to chance the setting is the master. Otherwise, if the setting is already enabled in the standby, it is very confusing to have it say that the setting must be enabled ... Authors: Ãlvaro Herrera, Petr Jelínek. Backpatch to 9.5. http://git.postgresql.org/pg/commitdiff/820ddb2c2f162ef1b086d870e9da0702834b6f32

Peter Eisentraut pushed:

Noah Misch pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Pavel Stěhule sent in two more revisions of a patch to add a custom function for converting human readable sizes to bytes.

Julien Rouhaud sent in a patch to make it possible to use hypothetical GIN indexes.

Kyotaro HORIGUCHI sent in a patch to enable asynchronous execution, which may speed up queries with long planning times.

David Rowley sent in two revisions of a patch to remove functionally dependent GROUP BY columns.

Rahila Syed sent in another revision of a patch to add a VACUUM progress checker.

Pavel Stěhule sent in another revision of a patch to implement ereport as a PL/PythonU function.

YUriy Zhuravlev sent in three more revisions of a patch to add some functionality to array slicing.

SAWADA Masahiko sent in three more revisions of a patch to add a "frozen" bit to the visibility map.

Daniel Verité sent in another revision of a patch to add \rotate to psql.

Pavel Stěhule, Robert Haas, and Michael Paquier traded patches to allow psql to handle multiple commands, expressed as multiple -c options, from the command line.

Artur Zakirov sent in another revision of a patch to improve Hunspell dictionary support in fulltext search.

Alexander Korotkov sent in another revision of a patch to use pg_rewind when target timeline was switched.

Tomas Vondra sent in a patch to add explicit fsync on the parent directory after all the rename() calls in timeline.c, xlog.c, xlogarchive.c and pgarch.c, and add START/END_CRIT_SECTION markers around the new fsync_fname calls (except for those in timeline.c, as the START/END_CRIT_SECTION is not available there).

Anastasia Lubennikova sent in another revision of a patch to add covering unique indexes.

Alexander Shulgin sent in a patch to create more stable query plans via more predictable column statistics.

Robert Haas sent in two more revisions of a patch to fix some infelicities between foreign join pushdown and EvalPlanQual.

Pavel Stěhule sent in another revision of a patch to implement parse_ident().

David Rowley sent in another revision of a patch to implement combining aggregates, which is foundational work for parallel aggregation.

Ashutosh Bapat sent in another revision of a patch to implement foreign join pushdown in the PostgreSQL FDW.

Julian Schauder and Craig Ringer traded patches to add 'waiting for replication' to pg_stat_activity.state.

Robert Haas sent in another revision of a patch to improve parallel explain and JOIN.

Haribabu Kommi sent in three more revisions of a patch to add a pg_hba_lookup function to get all matching pg_hba.conf entries.

Kyotaro HORIGUCHI sent in two more revisions of a patch to add a psql level filter for column outputs.

Kaigai Kouhei sent in another revision of a patch to add 'extnodename' fields in ForeignPath and ForeignScan, and embed ExtensibleNodeMethods in CustomPathMethods, CustomScanMethods and CustomExecMethods.

Michael Paquier sent in another revision of a patch to fix TAP checks.

Aleksander Alekseev sent in a patch to fix some bottlenecks for tables with many (in this case thousands of) partitions.

Michael Paquier sent in another revision of a patch to ensure consistent on-disk state of UNLOGGED indexes at recovery.

Amit Kapila sent in a patch to fix parallel buffer usage.

Stephen Frost sent in two more revisions of a patch to handle dependencies properly in ALTER POLICY.

Ildus Kurbangaliev sent in a patch to add support of partial decompression for datums.

David Rowley sent in a patch to add equivalence class filters.

Peter Geoghegan sent in another revision of a patch to quicksort when performing external sorts, use "tuple proper" memory pool for tuplesort merge, and perform memory prefetching when writing memtuples.

Peter Geoghegan sent in a patch to fix a documentation typo in INSERT in the exclusion constraints section.

par N Bougain le mercredi 9 décembre 2015 à 23h51

lundi 30 novembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 29 novembre 2015

PostgreSQL 9.5 Beta 2 disponible : http://www.postgresql.org/about/news/1625/

Les paquets RPM PostgreSQL 9.6devel sont disponibles pour les testeurs. Veillez à ne les utilisez que pour les crash tests : http://www.postgresql.org/message-id/1447660638.4285.45.camel@gunduz.org http://yum.PostgreSQL.org

FOSDEM PGDay est une conférence d'une journée qui sera tenue avant le FOSDEM à Bruxelles (Belgique) le 29 janvier 2015. Détails et appel à conférenciers ci-après : http://fosdem2016.pgconf.eu/

Prague PostgreSQL Developer Day 2016 (P2D2 2016) est une conférence sur deux jours, les 17 et 18 février 2016, à Prague (République Tchèque). Site en tchèque : http://www.p2d2.cz/

La première conférence PostgreSQL pan-asiatique se tiendra en mars 2016 à Singapour. L'appel à conférenciers est ouvert : http://2016.pgday.asia/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151130041315.GB22300@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Speed up ruleutils' name de-duplication code, and fix overlength-name case. Since commit 11e131854f8231a21613f834c40fe9d046926387, ruleutils.c has attempted to ensure that each RTE in a query or plan tree has a unique alias name. However, the code that was added for this could be quite slow, even as bad as O(N^3) if N identical RTE names must be replaced, as noted by Jeff Janes. Improve matters by building a transient hash table within set_rtable_names. The hash table in itself reduces the cost of detecting a duplicate from O(N) to O(1), and we can save another factor of N by storing the number of de-duplicated names already created for each entry, so that we don't have to re-try names already created. This way is probably a bit slower overall for small range tables, but almost by definition, such cases should not be a performance problem. In principle the same problem applies to the column-name-de-duplication code; but in practice that seems to be less of a problem, first because N is limited since we don't support extremely wide tables, and second because duplicate column names within an RTE are fairly rare, so that in practice the cost is more like O(N^2) not O(N^3). It would be very much messier to fix the column-name code, so for now I've left that alone. An independent problem in the same area was that the de-duplication code paid no attention to the identifier length limit, and would happily produce identifiers that were longer than NAMEDATALEN and wouldn't be unique after truncation to NAMEDATALEN. This could result in dump/reload failures, or perhaps even views that silently behaved differently than before. We can fix that by shortening the base name as needed. Fix it for both the relation and column name cases. In passing, check for interrupts in set_rtable_names, just in case it's still slow enough to be an issue. Back-patch to 9.3 where this code was introduced. http://git.postgresql.org/pg/commitdiff/8004953b5a2449c26c4e082771276b2f8629d153
  • Fix possible internal overflow in numeric division. div_var_fast() postpones propagating carries in the same way as mul_var(), so it has the same corner-case overflow risk we fixed in 246693e5ae8a36f0, namely that the size of the carries has to be accounted for when setting the threshold for executing a carry propagation step. We've not devised a test case illustrating the brokenness, but the required fix seems clear enough. Like the previous fix, back-patch to all active branches. Dean Rasheed http://git.postgresql.org/pg/commitdiff/5f10b7a604c87fc61a2c20a56552301f74c9bd5f
  • Accept flex > 2.5.x in configure. Per buildfarm member anchovy, 2.6.0 exists in the wild now. Hopefully it works with Postgres; if not, we'll have to do something about that, but in any case claiming it's "too old" is pretty silly. http://git.postgresql.org/pg/commitdiff/32f15d05c80044335f97347b5406f6736c06a033
  • Fix thinko: errmsg -> ereport. Silly mistake in my commit 09cecdf285ea9f51, reported by Erik Rijkers. The fact that the buildfarm didn't find this implies that we are not testing Perl builds that lack MULTIPLICITY, which is a bit disturbing from a coverage standpoint. Until today I'd have said nobody cared about such configurations anymore; but maybe not. http://git.postgresql.org/pg/commitdiff/9be3a4e24dc777e31f6358907ebefac841ea2632
  • Dodge a macro-name conflict with Perl. Some versions of Perl export a macro named HS_KEY. This creates a conflict in contrib/hstore_plperl against hstore's macro of the same name. The most future-proof solution seems to be to rename our macro; I chose HSTORE_KEY. For consistency, rename HS_VAL and related macros similarly. Back-patch to 9.5. contrib/hstore_plperl doesn't exist before that so there is no need to worry about the conflict in older releases. Per reports from Marco Atzeri and Mike Blackwell. http://git.postgresql.org/pg/commitdiff/68c1d7d42e553682f1d2723e623b6a3a4b02c75f
  • Fix handling of inherited check constraints in ALTER COLUMN TYPE (again). The previous way of reconstructing check constraints was to do a separate "ALTER TABLE ONLY tab ADD CONSTRAINT" for each table in an inheritance hierarchy. However, that way has no hope of reconstructing the check constraints' own inheritance properties correctly, as pointed out in bug #13779 from Jan Dirk Zijlstra. What we should do instead is to do a regular "ALTER TABLE", allowing recursion, at the topmost table that has a particular constraint, and then suppress the work queue entries for inherited instances of the constraint. Annoyingly, we'd tried to fix this behavior before, in commit 5ed6546cf, but we failed to notice that it wasn't reconstructing the pg_constraint field values correctly. As long as I'm touching pg_get_constraintdef_worker anyway, tweak it to always schema-qualify the target table name; this seems like useful backup to the protections installed by commit 5f173040. In HEAD/9.5, get rid of get_constraint_relation_oids, which is now unused. (I could alternatively have modified it to also return conislocal, but that seemed like a pretty single-purpose API, so let's not pretend it has some other use.) It's unused in the back branches as well, but I left it in place just in case some third-party code has decided to use it. In HEAD/9.5, also rename pg_get_constraintdef_string to pg_get_constraintdef_command, as the previous name did nothing to explain what that entry point did differently from others (and its comment was equally useless). Again, that change doesn't seem like material for back-patching. I did a bit of re-pgindenting in tablecmds.c in HEAD/9.5, as well. Otherwise, back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/074c5cfbfb4923158be9ccdb77420d6522d77538
  • Adopt the GNU convention for handling tar-archive members exceeding 8GB. The POSIX standard for tar headers requires archive member sizes to be printed in octal with at most 11 digits, limiting the representable file size to 8GB. However, GNU tar and apparently most other modern tars support a convention in which oversized values can be stored in base-256, allowing any practical file to be a tar member. Adopt this convention to remove two limitations: * pg_dump with -Ft output format failed if the contents of any one table exceeded 8GB. * pg_basebackup failed if the data directory contained any file exceeding 8GB. (This would be a fatal problem for installations configured with a table segment size of 8GB or more, and it has also been seen to fail when large core dump files exist in the data directory.) File sizes under 8GB are still printed in octal, so that no compatibility issues are created except in cases that would have failed entirely before. In addition, this patch fixes several bugs in the same area: * In 9.3 and later, we'd defined tarCreateHeader's file-size argument as size_t, which meant that on 32-bit machines it would write a corrupt tar header for file sizes between 4GB and 8GB, even though no error was raised. This broke both "pg_dump -Ft" and pg_basebackup for such cases. * pg_restore from a tar archive would fail on tables of size between 4GB and 8GB, on machines where either "size_t" or "unsigned long" is 32 bits. This happened even with an archive file not affected by the previous bug. * pg_basebackup would fail if there were files of size between 4GB and 8GB, even on 64-bit machines. * In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size, on 64-bit big-endian machines. In view of these potential data-loss bugs, back-patch to all supported branches, even though removal of the documented 8GB limit might otherwise be considered a new feature rather than a bug fix. http://git.postgresql.org/pg/commitdiff/00cdd83521cfdaaff0f566ebeadecc2cad4d51cf
  • Improve div_var_fast(), mostly by making comments better. The integer overflow situation in div_var_fast() is a great deal more complicated than the pre-existing comments would suggest. Moreover, the comments were also flat out incorrect as to the precise statement of the maxdiv loop invariant. Upon clarifying that, it becomes apparent that the way in which we updated maxdiv after a carry propagation pass was overly slow, complex, and conservative: we can just reset it to one, which is much easier and also reduces the number of times carry propagation occurs. Fix that and improve the relevant comments. Since this is mostly a comment fix, with only a rather marginal performance boost, no need for back-patch. Tom Lane and Dean Rasheed http://git.postgresql.org/pg/commitdiff/46166197c3b3748c3266c694d7c2f5a312ea928e
  • Be more paranoid about null return values from libpq status functions. PQhost() can return NULL in non-error situations, namely when a Unix-socket connection has been selected by default. That behavior is a tad debatable perhaps, but for the moment we should make sure that psql copes with it. Unfortunately, do_connect() failed to: it could pass a NULL pointer to strcmp(), resulting in crashes on most platforms. This was reported as a security issue by ChenQin of Topsec Security Team, but the consensus of the security list is that it's just a garden-variety bug with no security implications. For paranoia's sake, I made the keep_password test not trust PQuser or PQport either, even though I believe those will never return NULL given a valid PGconn. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/c5ef8ce53d37e276d70593ff0f4b06dd119cd3ff
  • Fix failure to consider failure cases in GetComboCommandId(). Failure to initially palloc the comboCids array, or to realloc it bigger when needed, left combocid's data structures in an inconsistent state that would cause trouble if the top transaction continues to execute. Noted while examining a user complaint about the amount of memory used for this. (There's not much we can do about that, but it does point up that repalloc failure has a non-negligible chance of occurring here.) In HEAD/9.5, also avoid possible invocation of memcpy() with a null pointer in SerializeComboCIDState; cf commit 13bba0227. http://git.postgresql.org/pg/commitdiff/0da3a9bef7ad36dc640aebf2d0482e18f21561f6
  • Improve PQhost() to return useful data for default Unix-socket connections. Previously, if no host information had been specified at connection time, PQhost() would return NULL (unless you are on Windows, in which case you got "localhost"). This is an unhelpful definition for a couple of reasons: it can cause corner-case crashes in applications (cf commit c5ef8ce53d), and there's no well-defined way for applications to find out the socket directory path that's actually in use. As an example of the latter problem, psql substituted DEFAULT_PGSOCKET_DIR for NULL in a couple of places, but this is subtly wrong because it's conceivable that psql is using a libpq shared library that was built with a different setting. Hence, change PQhost() to return DEFAULT_PGSOCKET_DIR when appropriate, and strip out the now-dead substitutions in psql. (There is still one remaining reference to DEFAULT_PGSOCKET_DIR in psql, in prompt.c, which I don't see a nice way to get rid of. But it only controls a prompt abbreviation decision, so it seems noncritical.) Also update the docs for PQhost, which had never previously mentioned the possibility of a socket directory path being returned. In passing fix the outright-incorrect code comment about PGconn.pgunixsocket. http://git.postgresql.org/pg/commitdiff/40cb21f70b4ef2721c38be6628298fb21fa7d2d2
  • Auto-generate file header comments in Unicode mapping files. Some of the Unicode/*.map files had identification comments added to them, evidently by hand. Others did not. Modify the generating scripts to produce these comments automatically, and update the generated files that lacked them. This is just minor cleanup as a by-product of trying to verify that the *.map files can indeed be reproduced from authoritative data. There are a depressingly large number that fail to reproduce from the claimed sources. I have not touched those in this commit, except for the JIS 2004-related files which required only a single comment update to match. Since this only affects comments, no need to consider a back-patch. http://git.postgresql.org/pg/commitdiff/e17dab53ea301031bf41d27e0799b940484c7bb0
  • Update UCS_to_GB18030.pl with info about origin of the reference file. http://git.postgresql.org/pg/commitdiff/5afdfc9cbb29ffc6f6b557a06495672d3c09f688
  • Avoid doing encoding conversions by double-conversion via MULE_INTERNAL. Previously, we did many conversions for Cyrillic and Central European single-byte encodings by converting to a related MULE_INTERNAL coding scheme before converting to the destination. This seems unnecessarily inefficient. Moreover, if the conversion encounters an untranslatable character, the error message will confusingly complain about failure to convert to or from MULE_INTERNAL, rather than the user-visible encodings. Worse still, this approach results in some completely unnecessary conversion failures; there are cases where the chosen MULE subset lacks characters that exist in both of the user-visible encodings, causing a conversion failure that need not occur. This patch fixes the first two of those deficiencies by introducing a new local2local() conversion support subroutine for direct conversion between any two single-byte character sets, and adding new conversion tables where needed. However, I generated the new conversion tables by testing PG 9.5's behavior, so that the actual conversion behavior is bug-compatible with previous releases; the only user-visible behavior change is that the error messages for conversion failures are saner. Changes in the conversion behavior will probably ensue after discussion. Interestingly, although this approach requires more tables, the .so files actually end up smaller (at least on my x86_64 machine); the tables are smaller than the management code needed for double conversion. Per a complaint from Albe Laurenz. http://git.postgresql.org/pg/commitdiff/8d32717b6bfaeda5b88b338dae728b47da19f4bb

Robert Haas pushed:

Peter Eisentraut pushed:

Andres Freund pushed:

Andrew Dunstan pushed:

Teodor Sigaev pushed:

Bruce Momjian pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Ãlvaro Herrera sent in a patch to improve cost estimates for BRIN indexes.

Ãlvaro Herrera sent in a patch to fix some misbehavior of the max_worker_processes GUC on standbys.

Peter Geoghegan sent in a patch to speed up CREATE INDEX CONCURRENTLY's TID sort.

Kyotaro HORIGUCHI sent in a patch to improve the documentation of floating point conversion for PL/pgsql.

Amit Kapila sent in two more revisions of a patch to speed up CLOG access.

Peter Eisentraut sent in a patch to play nicer with systemd.

Vitaly Burovoy sent in another revision of a patch to make fields extracted from 'infinity' timestamp[tz]s sane.

Victor Wagner sent in another revision of a patch to implement failover on the libpq connect level.

Amit Kapila sent in a patch to fix ExecParallelFinish() to be idempotent.

Ildus Kurbangaliev sent in two revisions of a patch to slice bufmgr into tranches similar to the way LWLocks are.

Robert Haas sent in a patch to tighten up how database names get checked for in pg_basebackup.

Nikolay Shaplov sent in two more revisions of a patch to add tuple data inspection to pageinspect.

Kyotaro HORIGUCHI sent in two revisions of a patch to add backend regexp processing to psql's tab completion implementation.

Ashutosh Bapat and Rushabh Lathia traded patches to get sorted data from a foreign server for merge join purposes.

SAWADA Masahiko sent in four more revisions of a patch to add a "frozen" bit to the visibility map.

Thomas Munro sent in another revision of a patch to add "causal reads."

Marko Tiikkaja sent in a patch to add a scale(numeric) function.

Marko Tiikkaja sent in a patch to add a trim(numeric) function.

Marko Tiikkaja sent in a PoC patch to implement "LISTEN *".

Amit Kapila sent in two revisions of a patch to fix an issue in parallel seq scan with parallel worker allocation.

Michael Paquier and Stephen Frost traded patches to add new default roles.

Robert Haas sent in another revision of a patch to implement parallel append.

Nikolay Shaplov sent in a patch to add a TAP test example.

Craig Ringer sent in another revision of a patch to implement pg_logical, a general-purpose logical output plugin.

Marko Tiikkaja sent in two more revisions of a patch to add a num_nulls() function.

Marko Tiikkaja sent in another revision of a patch to add a single_value() aggregate.

Taiki Kondo and Kyotaro HORIGUCHI traded patches to improve query performance by doing join pushdowns to table partitions.

Michael Paquier sent in two more revisions of a patch to add a dedicated WAL record forcing fsync().

Andreas Karlsson sent in another revision of a patch to cause a SIGHUP sent to the backend to reload SSL certificates.

Amit Langote sent in two revisions of a patch to fix some comments in nodeGather.c.

Haribabu Kommi sent in another revision of a patch to add a pg_hba_lookup function to get all matching pg_hba.conf entries.

Pavel Stěhule sent in another revision of a patch to add a custom function for converting human readable sizes to bytes.

Michael Paquier and Ãlvaro Herrera traded patches to add in-core regression tests for replication, cascading, archiving, PITR, etc.

David Steele sent in a patch to enable client log output filtering.

Catalin Iacob sent in a patch to update the docs for the case when psql is given multiple -c commands.

Pavel Stěhule sent in another revision of a patch to implement multiple -c commands in psql.

Robert Haas sent in a patch to allow the optimizer to generate plans where a Nested Loop or Hash Join appears below a Gather node and improve the output of EXPLAIN when parallel workers are used.

Magnus Hagander and Michael Paquier traded patches to initialize more parts of the walsnd structure.

Kaigai Kouhei sent in a patch to adds a Path *fdw_outerpath field to the ForeignPath node.

Dean Rasheed sent in another revision of a patch to add trig functions which take degrees as input.

Dmitry Ivanov sent in a patch to add a dump_stat extension which, as implied in the name, allows dumping the contents of pg_statistic in a form that lets it be reloaded.

Peter Geoghegan sent in a patch to ensure that GatherPath is within print_path(). Not having it there prevented complete information from appearing when using OPTIMIZER_DEBUG.

Tomas Vondra sent in some patches to fix a possible issue with fsync() not actually being issued at the appropriate times.

Noah Misch sent in a patch to clarify some comments in src/backend/access/transam/multixact.c.

Pavel Stěhule sent in another revision of a patch to add an ereport() function to PL/PythonU.

Peter Geoghegan sent in a patch to abort C collation text abbreviation less frequently.

par N Bougain le lundi 30 novembre 2015 à 22h19

jeudi 19 novembre 2015

Guillaume Lelarge

Version finale du livre

Elle n'est pas encore sortie. Elle est pratiquement terminée, on attend d'avoir le livre en version imprimée.

Néanmoins, je peux déjà dire les nouveautés par rapport à la beta 0.4 :

  • Global
    • mise à jour du texte pour la 9.5
    • ajout du chapitre sur la sécurité
    • ajout du chapitre sur la planification
    • mise à jour des exemples avec PostgreSQL 9.5 beta 1
  • Fichiers
    • Ajout d'un schéma sur les relations entre tables, FSM et VM
    • Ajout de la description des répertoires pg_dynshmem et pg_logical
  • Contenu des fichiers
    • Ajout d'informations sur le stockage des données, colonne par colonne
    • Ajout d'un schéma sur la structure logique et physique d'un index B-tree
    • Ajout de la description des index GIN
    • Ajout de la description des index GiST
    • Ajout de la description des index SP-GiST
  • Architecture mémoire
    • calcul du work_mem pour un tri
    • calcul du maintenance_work_mem pour un VACUUM
  • Gestion des transactions
    • Gestion des verrous et des accès concurrents
  • Maintenance
    • Description de la sortie d'un VACUUM VERBOSE

J'avoue que j'ai hâte d'avoir la version finale entre mes mains :-) Bah, oui, c'est quand même 1 an et demi de boulot acharné !

par Guillaume Lelarge le jeudi 19 novembre 2015 à 22h36

mardi 17 novembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 15 novembre 2015

PostgreSQL 9.5 Beta 2 disponible. À vos tests ! http://www.postgresql.org/about/news/1625/

[ndt: Stand PostgreSQL au Paris Open Source Summit ces 18 & 19 novembre]

[ndt: Rappel : AG PostgreSQL.fr ce mercredi 18 au soir]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151116070119.GB23214@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Andres Freund pushed:

Robert Haas pushed:

  • Add a dummy return statement to TupleQueueRemap. This is unreachable for multiple reasons, but per Amit Kapila the Windows compiler he is using still thinks we can get there. http://git.postgresql.org/pg/commitdiff/89ff5c7f7512a919e76407b4f8269e5b0b39dd2b
  • Fix rebasing mistake in nodeGather.c The patches committed as 6e71dd7ce9766582da453f493bc371d64977282f and 3a1f8611f2582df0a16bcd35caed2e1526387643 were developed in parallel but dependent on each other in a way that I failed to notice. This patch to fix the problem was prepared by Amit Kapila. http://git.postgresql.org/pg/commitdiff/bf3d0156311f0d8fa7705d8de0d07c4e0298051a
  • Comment update. Adjust to account for 5fc4c26db5120bd90348b6ee3101fcddfdf54800. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/5c90a2ffddd2504a7441e80a22ba0fe8d484291a
  • Add outfuncs.c support for GatherPath. I dunno how commit 3bd909b220930f21d6e15833a17947be749e7fde missed this, but it evidently did. http://git.postgresql.org/pg/commitdiff/f764ecd81b2a8a1e9000d43a73ca5eec8e8008bc
  • Make sequential scans parallel-aware. In addition, this path fills in a number of missing bits and pieces in the parallel infrastructure. Paths and plans now have a parallel_aware flag indicating whether whatever parallel-aware logic they have should be engaged. It is believed that we will need this flag for a number of path/plan types, not just sequential scans, which is why the flag is generic rather than part of the SeqScan structures specifically. Also, execParallel.c now gives parallel nodes a chance to initialize their PlanState nodes from the DSM during parallel worker startup. Amit Kapila, with a fair amount of adjustment by me. Review of previous patch versions by Haribabu Kommi and others. http://git.postgresql.org/pg/commitdiff/f0661c4e8c44c0ec7acd4ea7c82e85b265447398
  • Generate parallel sequential scan plans in simple cases. Add a new flag, consider_parallel, to each RelOptInfo, indicating whether a plan for that relation could conceivably be run inside of a parallel worker. Right now, we're pretty conservative: for example, it might be possible to defer applying a parallel-restricted qual in a worker, and later do it in the leader, but right now we just don't try to parallelize access to that relation. That's probably the right decision in most cases, anyway. Using the new flag, generate parallel sequential scan plans for plain baserels, meaning that we now have parallel sequential scan in PostgreSQL. The logic here is pretty unsophisticated right now: the costing model probably isn't right in detail, and we can't push joins beneath Gather nodes, so the number of plans that can actually benefit from this is pretty limited right now. Lots more work is needed. Nevertheless, it seems time to enable this functionality so that all this code can actually be tested easily by users and developers. Note that, if you wish to test this functionality, it will be necessary to set max_parallel_degree to a value greater than the default of 0. Once a few more loose ends have been tidied up here, we might want to consider changing the default value of this GUC, but I'm leaving it alone for now. Along the way, fix a bug in cost_gather: the previous coding thought that a Gather node's transfer overhead should be costed on the basis of the relation size rather than the number of tuples that actually need to be passed off to the leader. Patch by me, reviewed in earlier versions by Amit Kapila. http://git.postgresql.org/pg/commitdiff/80558c1f5aa109d08db0fbd76a6d370f900628a8
  • Provide readfuncs support for custom scans. Commit a0d9f6e434bb56f7e5441b7988f3982feead33b3 added this support for all other plan node types; this fills in the gap. Since TextOutCustomScan complicates this and is pretty well useless, remove it. KaiGai Kohei, with some modifications by me. http://git.postgresql.org/pg/commitdiff/a05dc4d7fd57d4ae084c1f0801973e5c1a1aa26e
  • Make idle backends exit if the postmaster dies. Letting backends continue to run if the postmaster has exited prevents PostgreSQL from being restarted, which in many environments is catastrophic. Worse, if some other backend crashes, we no longer have any protection against shared memory corruption. So, arrange for them to exit instead. We don't want to expend many cycles on this, but including postmaster death in the set of things that we wait for when a backend is idle seems cheap enough. Rajeev Rastogi and Robert Haas http://git.postgresql.org/pg/commitdiff/ac1d7945f866b1928c2554c0f80fd52d7f977772
  • Move each SLRU's lwlocks to a separate tranche. This makes it significantly easier to identify these lwlocks in LWLOCK_STATS or Trace_lwlocks output. It's also arguably better from a modularity standpoint, since lwlock.c no longer needs to know anything about the LWLock needs of the higher-level SLRU facility. Ildus Kurbangaliev, reviewd by Ãlvaro Herrera and by me. http://git.postgresql.org/pg/commitdiff/fe702a7b3f9f2bc5bf6d173166d7d55226af82c8
  • libpq: Notice errors a backend may have sent just before dying. At least since the introduction of Hot Standby, the backend has sometimes sent fatal errors even when no client query was in progress, assuming that the client would receive it. However, pqHandleSendFailure was not in sync with this assumption, and only tries to catch notices and notifies. Add a parseInput call to the loop there to fix. Andres Freund suggested the fix. Comments are by me. Reviewed by Michael Paquier. http://git.postgresql.org/pg/commitdiff/c3e7c24a1d60dc6ad56e2a0723399f1570c54224
  • Remove accidentally-committed debugging code. Amit Kapila http://git.postgresql.org/pg/commitdiff/179c97bf587df710f98aa3ce2d95f503b0757e15

Tom Lane pushed:

  • Improve our workaround for 'TeX capacity exceeded' in building PDF files. In commit a5ec86a7c787832d28d5e50400ec96a5190f2555 I wrote a quick hack that reduced the number of TeX string pool entries created while converting our documentation to PDF form. That held the fort for awhile, but as of HEAD we're back up against the same limitation. It turns out that the original coding of \FlowObjectSetup actually results in *three* string pool entries being generated for every "flow object" (that is, potential cross-reference target) in the documentation, and my previous hack only got rid of one of them. With a little more care, we can reduce the string count to one per flow object plus one per actually-cross-referenced flow object (about 115000 + 5000 as of current HEAD); that should work until the documentation volume roughly doubles from where it is today. As a not-incidental side benefit, this change also causes pdfjadetex to stop emitting unreferenced hyperlink anchors (bookmarks) into the PDF file. It had been making one willy-nilly for every flow object; now it's just one per actually-cross-referenced object. This results in close to a 2X savings in PDF file size. We will still want to run the output through "jpdftweak" to get it to be compressed; but we no longer need removal of unreferenced bookmarks, so we might be able to find a quicker tool for that step. Although the failure only affects HEAD and US-format output at the moment, 9.5 cannot be more than a few pages short of failing likewise, so it will inevitably fail after a few rounds of minor-version release notes. I don't have a lot of faith that we'll never hit the limit in the older branches; and anyway it would be nice to get rid of jpdftweak across the board. Therefore, back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/944b41fc00600b74f518005ac314cc222bf6abd5
  • Add missing "static" qualifier. Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/b05ae27e9a57a1c8f1f44720c23c4828106b7205
  • Docs: fix misleading example. Commit 8457d0beca731bf0 introduced an example which, while not incorrect, failed to exhibit the behavior it meant to describe, as a result of omitting an E'' prefix that needed to be there. Noticed and fixed by Peter Geoghegan. I (tgl) failed to resist the temptation to wordsmith nearby text a bit while at it. http://git.postgresql.org/pg/commitdiff/7b6fb76349fccc38f0ef96166f207c6acd21968c
  • Be more noisy about "wrong number of nailed relations" initfile problems. In commit 5d1ff6bd559ea8df1b7302e245e690b01b9a4fa4 I added some logic to relcache.c to try to ensure that the regression tests would fail if we made a mistake about which relations belong in the relcache init files. I'm quite sure I tested that, but I must have done so only for the non-shared-catalog case, because a report from Adam Brightwell showed that the regression tests still pass just fine if we bollix the shared-catalog init file in the way this code was supposed to catch. The reason is that that file gets loaded before we do client authentication, so the WARNING is not sent to the client, only to the postmaster log, where it's far too easily missed. The least Rube Goldbergian answer to this is to put an Assert(false) after the elog(WARNING). That will certainly get developers' attention, while not breaking production builds' ability to recover from corner cases with similar symptoms. Since this is only of interest to developers, there seems no need for a back-patch, even though the previous commit went into all branches. http://git.postgresql.org/pg/commitdiff/da3751c8ea6f833d8e69fbdd168d29d328674398
  • Improve documentation around autovacuum-related storage parameters. These were discussed in three different sections of the manual, which unsurprisingly had diverged over time; and the descriptions of individual variables lacked stylistic consistency even within each section (and frequently weren't in very good English anyway). Clean up the mess, and remove some of the redundant information in hopes that future additions will be less likely to re-introduce inconsistency. For instance I see no need for maintenance.sgml to include its very own list of all the autovacuum storage parameters, especially since that list was already incomplete. http://git.postgresql.org/pg/commitdiff/6404751ce91fb02999e46700a1ac62b3c6319abf
  • Do a round of copy-editing on the 9.5 release notes. Also fill in the previously empty "major enhancements" list. YMMV as to which items should make the cut, but it's past time we had something more than a placeholder here. (I meant to get this done before beta2 was wrapped, but got distracted by PDF build problems. Better late than never.) http://git.postgresql.org/pg/commitdiff/39b9978d9cac34ad799a5fa3ff3846f3e0372b0a
  • Fix unwanted flushing of libpq's input buffer when socket EOF is seen. In commit 210eb9b743c0645d I centralized libpq's logic for closing down the backend communication socket, and made the new pqDropConnection routine always reset the I/O buffers to empty. Many of the call sites previously had not had such code, and while that amounted to an oversight in some cases, there was one place where it was intentional and necessary *not* to flush the input buffer: pqReadData should never cause that to happen, since we probably still want to process whatever data we read. This is the true cause of the problem Robert was attempting to fix in c3e7c24a1d60dc6a, namely that libpq no longer reported the backend's final ERROR message before reporting "server closed the connection unexpectedly". But that only accidentally fixed it, by invoking parseInput before the input buffer got flushed; and very likely there are timing scenarios where we'd still lose the message before processing it. To fix, pass a flag to pqDropConnection to tell it whether to flush the input buffer or not. On review I think flushing is actually correct for every other call site. Back-patch to 9.3 where the problem was introduced. In HEAD, also improve the comments added by c3e7c24a1d60dc6a. http://git.postgresql.org/pg/commitdiff/c405918858c09d4264ffc5a3c73d57dc3efb0213
  • Improve type numeric's calculations for ln(), log(), exp(), pow(). Set the "rscales" for intermediate-result calculations to ensure that suitable numbers of significant digits are maintained throughout. The previous coding hadn't thought this through in any detail, and as a result could deliver results with many inaccurate digits, or in the worst cases even fail with divide-by-zero errors as a result of losing all nonzero digits of intermediate results. In exp_var(), get rid entirely of the logic that separated the calculation into integer and fractional parts: that was neither accurate nor particularly fast. The existing range-reduction method of dividing by 2^n can be applied across the full input range instead of only 0..1, as long as we are careful to set an appropriate rscale for each step. Also fix the logic in mul_var() for shortening the calculation when the caller asks for fewer output digits than an exact calculation would require. This bug doesn't affect simple multiplications since that code path asks for an exact result, but it does contribute to accuracy issues in the transcendental math functions. In passing, improve performance of mul_var() a bit by forcing the shorter input to be on the left, thus reducing the number of iterations of the outer loop and probably also reducing the number of carry-propagation steps needed. This is arguably a bug fix, but in view of the lack of field complaints, it does not seem worth the risk of back-patching. Dean Rasheed http://git.postgresql.org/pg/commitdiff/7d9a4737c268f61fb8800957631f12d3f13be218
  • Fix ruleutils.c's dumping of whole-row Vars in ROW() and VALUES() contexts. Normally ruleutils prints a whole-row Var as "foo.*". We already knew that that doesn't work at top level of a SELECT list, because the parser would treat the "*" as a directive to expand the reference into separate columns, not a whole-row Var. However, Joshua Yanovski points out in bug #13776 that the same thing happens at top level of a ROW() construct; and some nosing around in the parser shows that the same is true in VALUES(). Hence, apply the same workaround already devised for the SELECT-list case, namely to add a forced cast to the appropriate rowtype in these cases. (The alternative of just printing "foo" was rejected because it is difficult to avoid ambiguity against plain columns named "foo".) Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/7745bc352a82bd588be986479c7aabc3b076a375

Ãlvaro Herrera pushed:

  • vacuumdb: don't prompt for passwords over and over. Having the script prompt for passwords over and over was a preexisting problem when it processed multiple databases or when it processed multiple analyze stages, but the parallel mode introduced in commit a179232047 made it worse. Fix the annoyance by keeping a copy of the password used by the first connection that requires one. Since users can (currently) only have a single password, there's no need for more complex arrangements (such as remembering one password per database). Per bug #13741 reported by Eric Brown. Patch authored and cross-reviewed by Haribabu Kommi and Michael Paquier, slightly tweaked by Ãlvaro Herrera. Discussion: http://www.postgresql.org/message-id/20151027193919.931.54948@wrigleys.postgresql.org Backpatch to 9.5, where parallel vacuumdb was introduced. http://git.postgresql.org/pg/commitdiff/83dec5a712af251af15effbf781ddaedc3bf6b3b

Stephen Frost pushed:

Bruce Momjian pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Adam Brightwell sent in two revisions of a patch to bootstrap pg_shseclabel in relcache initialization.

Kaigai Kouhei sent in another revision of a patch to reconcile Foreign join pushdown with EvalPlanQual.

Ashutosh Bapat sent in another revision of a patch to add a transaction manager for foreign servers.

Marco Nenciarini sent in a patch to fix an incompatibility between 9.3's WAL stream and pg_receivelog.

Kaigai Kouhei sent in a patch for each of 9.5 and git master to allow CustomScan on copyfuncs.

Pavel Stěhule sent in another revision of a patch to allow psql to send multiple SQL statements via the command line.

Pavel Stěhule and Catalin Iacob traded patches to add an ereport function to PL/PythonU.

Artur Zakirov sent in another revision of a patch to enhance the performance of Hunspell dictionaries in full-text search.

Peter Geoghegan sent in a patch to fix some erroneous documentation of char(n).

Michael Paquier sent in another revision of a patch to add trigonometric functions which take degree values as input.

Andres Freund sent in another revision of a patch to control continuous flushing of checkpoints.

Thomas Munro sent in two revisions of a patch to create a "causal reads" capability.

Peter Eisentraut sent in another revision of a patch to handle missing pieces of compilation infrastructure more gracefully.

Michael Paquier sent in another revision of a patch to show tuple data in pageinspect.

Craig Ringer sent in a patch to implement pg_logical, a plugin based on logical WAL streams.

Kyotaro HORIGUCHI sent in a patch to allow using regexes in frontend code, in particular tab completion for psql.

Thomas Munro sent in another revision of a patch to simplify tab completion in psql.

Alexander Korotkov sent in another revision of a patch to rework the access method interface.

Peter Geoghegan sent in a patch to expand SortSupport to char(n), bytea, and alternative opclasses.

Kaigai Kouhei sent in a patch to add serialization support to CustomScan.

SAWADA Masahiko sent in two more revisions of a patch to add a "frozen" bit to the visibility map.

Amit Kapila sent in a patch to fix an issue in the parallel seq scan code.

Robert Haas sent in a patch to implement Append in parallel.

Jeff Janes and Tom Lane traded patches to speed up set_rtable_names.

Amit Kapila sent in a patch to fix some issues in parallel seq scan.

Robert Haas sent in a patch to tranche-ify buffer locks.

Haribabu Kommi sent in another revision of a patch to skip ALTER x SET SCHEMA if the schema didn't change.

par N Bougain le mardi 17 novembre 2015 à 22h47

jeudi 12 novembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 8 novembre 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151108233714.GA15118@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Kevin Grittner pushed:

Robert Haas pushed:

  • Fix problems with ParamListInfo serialization mechanism. Commit d1b7c1ffe72e86932b5395f29e006c3f503bc53d introduced a mechanism for serializing a ParamListInfo structure to be passed to a parallel worker. However, this mechanism failed to handle external expanded values, as pointed out by Noah Misch. Repair. Moreover, plpgsql_param_fetch requires adjustment because the serialization mechanism needs it to skip evaluating unused parameters just as we would do when it is called from copyParamList, but params == estate->paramLI in that case. To fix, make the bms_is_member test in that function unconditional. Finally, have setup_param_list set a new ParamListInfo field, paramMask, to the parameters actually used in the expression, so that we don't try to fetch those that are not needed when serializing a parameter list. This isn't necessary for correctness, but it makes the performance of the parallel executor code comparable to what we do for cases involving cursors. Design suggestions and extensive review by Noah Misch. Patch by me. http://git.postgresql.org/pg/commitdiff/1efc7e538204646fec0f790b683012ed9bd8a99f
  • Correct tiny inaccuracy in strxfrm cache comment. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/0279f62fdca7ff19ef34a836f5201935dc7f627c
  • shm_mq: Third attempt at fixing nowait behavior in shm_mq_receive. Commit a1480ec1d3bacb9acb08ec09f22bc25bc033115b purported to fix the problems with commit b2ccb5f4e6c81305386edb34daf7d1d1e1ee112a, but it didn't completely fix them. The problem is that the checks were performed in the wrong order, leading to a race condition. If the sender attached, sent a message, and detached after the receiver called shm_mq_get_sender and before the receiver called shm_mq_counterparty_gone, we'd incorrectly return SHM_MQ_DETACHED before all messages were read. Repair by reversing the order of operations, and add a long comment explaining why this new logic is (hopefully) correct. http://git.postgresql.org/pg/commitdiff/4efe26cbd3ef0d85656bf00ac9e5bd41cc8a2a36
  • shm_mq: Third attempt at fixing nowait behavior in shm_mq_receive. Commit a1480ec1d3bacb9acb08ec09f22bc25bc033115b purported to fix the problems with commit b2ccb5f4e6c81305386edb34daf7d1d1e1ee112a, but it didn't completely fix them. The problem is that the checks were performed in the wrong order, leading to a race condition. If the sender attached, sent a message, and detached after the receiver called shm_mq_get_sender and before the receiver called shm_mq_counterparty_gone, we'd incorrectly return SHM_MQ_DETACHED before all messages were read. Repair by reversing the order of operations, and add a long comment explaining why this new logic is (hopefully) correct. http://git.postgresql.org/pg/commitdiff/4efe26cbd3ef0d85656bf00ac9e5bd41cc8a2a36
  • Improve comments about abbreviation abort. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/ee44cb7566ffafc0144535e1f966f5e3bb7d384b
  • Pass extra data to bgworkers, and use this to fix parallel contexts. Up until now, the total amount of data that could be passed to a background worker at startup was one datum, which can be a small as 4 bytes on some systems. That's enough to pass a dsm_handle or an array index, but not much else. Add a bgw_extra flag to the BackgroundWorker struct, allowing up to 128 bytes to be passed to a new worker on any platform. Use this to fix a problem I recently discovered with the parallel context machinery added in 9.5: the master assigns each worker an array index, and each worker subsequently assigns itself an array index, and there's nothing to guarantee that the two sets of indexes match, leading to chaos. Normally, I would not back-patch the change to add bgw_extra, since it is basically a feature addition. However, since 9.5 is still in beta and there seems to be no other sensible way to repair the broken parallel context machinery, back-patch to 9.5. Existing background worker code can ignore the bgw_extra field without a problem, but might need to be recompiled since the structure size has changed. Report and patch by me. Review by Amit Kapila. http://git.postgresql.org/pg/commitdiff/64b2e7ad917a9a7814904d0f6dbde52cefbcfa00
  • Document interaction of bgworkers with LISTEN/NOTIFY. Thomas Munro and Robert Haas, reviewed by Haribabu Kommi http://git.postgresql.org/pg/commitdiff/dde5f09fad3ac188a155e21667f76825f43a28c8
  • pg_size_pretty: Format negative values similar to positive ones. Previously, negative values were always displayed in bytes, regardless of how large they were. Adrian Vondendriesch, reviewed by Julien Rouhaud and myself http://git.postgresql.org/pg/commitdiff/8a1fab36aba7506fcf4559c4ef95fcacdd0b439a
  • Add sort support routine for the UUID data type. This introduces a simple encoding scheme to produce abbreviated keys: pack as many bytes of each UUID as will fit into a Datum. On little-endian machines, a byteswap is also performed; the abbreviated comparator can therefore just consist of a simple 3-way unsigned integer comparison. The purpose of this change is to speed up sorting data on a column of type UUID. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/a76ef15d9fc9207a0758e8d6f6700dc8c931a934
  • Remove unnecessary cast in previous commit. Noted by Kyotaro Horiguchi, who also reviewed the previous patch, but I failed to notice his review before committing. http://git.postgresql.org/pg/commitdiff/cbb82e370da8e80aa9667ffbb395cabb0529ea00
  • When completing ALTER INDEX .. SET, add an equals sign also. Jeff Janes http://git.postgresql.org/pg/commitdiff/af9773cf4cac235a32341b11864c4a0a9a22b6c1
  • Try to convince gcc that TupleQueueRemap never falls off the end. Without this, MacOS gcc version 4.2.1 isn't convinced. http://git.postgresql.org/pg/commitdiff/8d7396e5092f1bc6433a38ccabaf88c8c681de11
  • Modify tqueue infrastructure to support transient record types. Commit 4a4e6893aa080b9094dadbe0e65f8a75fee41ac6, which introduced this mechanism, failed to account for the fact that the RECORD pseudo-type uses transient typmods that are only meaningful within a single backend. Transferring such tuples without modification between two cooperating backends does not work. This commit installs a system for passing the tuple descriptors over the same shm_mq being used to send the tuples themselves. The two sides might not assign the same transient typmod to any given tuple descriptor, so we must also substitute the appropriate receiver-side typmod for the one used by the sender. That adds some CPU overhead, but still seems better than being unable to pass records between cooperating parallel processes. Along the way, move the logic for handling multiple tuple queues from tqueue.c to nodeGather.c; tqueue.c now provides a TupleQueueReader, which reads from a single queue, rather than a TupleQueueFunnel, which potentially reads from multiple queues. This change was suggested previously as a way to make sure that nodeGather.c rather than tqueue.c had policy control over the order in which to read from queues, but it wasn't clear to me until now how good an idea it was. typmod mapping needs to be performed separately for each queue, and it is much simpler if the tqueue.c code handles that and leaves multiplexing multiple queues to higher layers of the stack. http://git.postgresql.org/pg/commitdiff/6e71dd7ce9766582da453f493bc371d64977282f
  • Remove set-but-not-used variables. Reported by both Peter Eisentraunt and Kevin Grittner. http://git.postgresql.org/pg/commitdiff/fba60e573e092991dde4f757b88384ab1ec6a0ce

Tom Lane pushed:

  • Remove some more dead Alpha-specific code. http://git.postgresql.org/pg/commitdiff/620ac88d6fd39c01bd25c90491333f088d186151
  • Code + docs review for unicode linestyle patch. Fix some brain fade in commit a2dabf0e1dda93c8: erroneous variable names in docs, rearrangements that made sentences less clear not more so, undocumented and poorly-chosen-anyway API behaviors of subroutines, bad grammar in error messages, copy-and-paste faults. Albe Laurenz and Tom Lane http://git.postgresql.org/pg/commitdiff/a69b0b2c14c614c129e89ae96d6f624375531760
  • Remove obsolete advice about doubling backslashes in regex escapes. Standard-conforming literals have been the default for long enough that it no longer seems necessary to go out of our way to tell people to write regex escapes illegibly. http://git.postgresql.org/pg/commitdiff/fc0b8935213cda555d2b3af2ed08da28ed120e31
  • Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me http://git.postgresql.org/pg/commitdiff/d89494166351e1fdac77d87c6af500401deb2422
  • Add regression tests for remote execution of extension operators/functions. Rather than relying on other extensions to be available for installation, let's just add some test objects to the postgres_fdw extension itself within the regression script. http://git.postgresql.org/pg/commitdiff/b9f117d6cd3c79780b0c0e57068f0837f7493aa6
  • Improve implementation of GEQO's init_tour() function. Rather than filling a temporary array and then copying values to the output array, we can generate the required random permutation in-place using the Fisher-Yates shuffle algorithm. This is shorter as well as more efficient than before. It's pretty unlikely that anyone would notice a speed improvement, but shorter code is better. Nathan Wagner, edited a bit by me http://git.postgresql.org/pg/commitdiff/59464bd6f928ad0da30502cbe9b54baec9ca2c69
  • Fix memory leaks in PL/Python. Previously, plpython was in the habit of allocating a lot of stuff in TopMemoryContext, and it was very slipshod about making sure that stuff got cleaned up; in particular, use of TopMemoryContext as fn_mcxt for function calls represents an unfixable leak, since we generally don't know what the called function might have allocated in fn_mcxt. This results in session-lifespan leakage in certain usage scenarios, as for example in a case reported by Ed Behn back in July. To fix, get rid of all the retail allocations in TopMemoryContext. All long-lived allocations are now made in sub-contexts that are associated with specific objects (either pl/python procedures, or Python-visible objects such as cursors and plans). We can clean these up when the associated object is deleted. I went so far as to get rid of PLy_malloc completely. There were a couple of places where it could still have been used safely, but on the whole it was just an invitation to bad coding. Haribabu Kommi, based on a draft patch by Heikki Linnakangas; some further work by me http://git.postgresql.org/pg/commitdiff/8c75ad436f75fc629b61f601ba884c8f9313c9af
  • Fix erroneous hash calculations in gin_extract_jsonb_path(). The jsonb_path_ops code calculated hash values inconsistently in some cases involving nested arrays and objects. This would result in queries possibly not finding entries that they should find, when using a jsonb_path_ops GIN index for the search. The problem cases involve JSONB values that contain both scalars and sub-objects at the same nesting level, for example an array containing both scalars and sub-arrays. To fix, reset the current stack->hash after processing each value or sub-object, not before; and don't try to be cute about the outermost level's initial hash. Correcting this means that existing jsonb_path_ops indexes may now be inconsistent with the new hash calculation code. The symptom is the same --- searches not finding entries they should find --- but the specific rows affected are likely to be different. Users will need to REINDEX jsonb_path_ops indexes to make sure that all searches work as expected. Per bug #13756 from Daniel Cheng. Back-patch to 9.4 where the faulty logic was introduced. http://git.postgresql.org/pg/commitdiff/b23af458755ed90ae9164857e59d034bad75a672
  • Fix enforcement of restrictions inside regexp lookaround constraints. Lookahead and lookbehind constraints aren't allowed to contain backrefs, and parentheses within them are always considered non-capturing. Or so says the manual. But the regexp parser forgot about these rules once inside a parenthesized subexpression, so that constructs like (\w)(?=(\1)) were accepted (but then not correctly executed --- a case like this acted like (\w)(?=\w), without any enforcement that the two \w's match the same text). And in (?=((foo))) the innermost parentheses would be counted as capturing parentheses, though no text would ever be captured for them. To fix, properly pass down the "type" argument to the recursive invocation of parse(). Back-patch to all supported branches; it was agreed that silent misexecution of such patterns is worse than throwing an error, even though new errors in minor releases are generally not desirable. http://git.postgresql.org/pg/commitdiff/a43b4ab1111ca5e5f40a2ddd8e56bf999b9fdad9
  • Rename PQsslAttributes() to PQsslAttributeNames(), and const-ify fully. Per discussion, the original name was a bit misleading, and PQsslAttributeNames() seems more apropos. It's not quite too late to change this in 9.5, so let's change it while we can. Also, make sure that the pointer array is const, not only the pointed-to strings. Minor documentation wordsmithing while at it. Lars Kanis, slight adjustments by me http://git.postgresql.org/pg/commitdiff/9042f583429ef880665dd36b7bcad45ba62dea7b
  • Add "xid <> xid" and "xid <> int4" operators. The corresponding "=" operators have been there a long time, and not having their negators is a bit of a nuisance. Michael Paquier http://git.postgresql.org/pg/commitdiff/c5e86ea932794051d2fe3bd26ce4c03981b0b826
  • Update 9.5 release notes through today. http://git.postgresql.org/pg/commitdiff/ad9fad7b68c4209db7d73f59a538ffe619c07872

Peter Eisentraut pushed:

Stephen Frost pushed:

  • Set include_realm=1 default in parse_hba_line. With include_realm=1 being set down in parse_hba_auth_opt, if multiple options are passed on the pg_hba line, such as: host all all 0.0.0.0/0 gss include_realm=0 krb_realm=XYZ.COM We would mistakenly reset include_realm back to 1. Instead, we need to set include_realm=1 up in parse_hba_line, prior to parsing any of the additional options. Discovered by Jeff McCormick during testing. Bug introduced by 9a08841. Back-patch to 9.5 http://git.postgresql.org/pg/commitdiff/5644419b3de418f81e4461cbaa06c5d4b5da0800

Noah Misch pushed:

  • Don't connect() to a wildcard address in test_postmaster_connection(). At least OpenBSD, NetBSD, and Windows don't support it. This repairs pg_ctl for listen_addresses='0.0.0.0' and listen_addresses='::'. Since pg_ctl prefers to test a Unix-domain socket, Windows users are most likely to need this change. Back-patch to 9.1 (all supported versions). This could change pg_ctl interaction with loopback-interface firewall rules. Therefore, in 9.4 and earlier (released branches), activate the change only on known-affected platforms. Reported (bug #13611) and designed by Kondo Yuta. http://git.postgresql.org/pg/commitdiff/fed19f312c72778ce7dcbb0670871b22bd06ff85

Andres Freund pushed:

  • Set replication origin when decoding commit records. By accident the replication origin was not set properly in DecodeCommit(). That's bad because the origin is passed to the output plugins origin filter, and accessible from the output plugin via ReorderBufferTXN->origin_id. Accessing the origin of individual changes worked before the fix, which is why this wasn't notices earlier. Reported-By: Craig Ringer Author: Craig Ringer Discussion: CAMsr+YFhBJLp=qfSz3-J+0P1zLkE8zNXM2otycn20QRMx380gw@mail.gmail.com Backpatch: 9.5, where replication origins where introduced http://git.postgresql.org/pg/commitdiff/f3a764b0dac77451d20ee00d2f5916447a5c4346

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Alexander Korotkov sent in another revision of a patch to make the access method API more extensible.

David Fetter sent in a patch to add weighted statistics to PostgreSQL.

Craig Ringer sent in a patch to add a logical decoding output plugin.

Robert Haas and Rajeev Rastogi traded patches to fix an issue that could result in a dangling client backend process.

Peter Geoghegan sent in a patch to correct a heap_finish_speculative() error message inaccuracy.

Bill Moran sent in a patch to add new configuration options to TOAST.

Pavel Stěhule sent in another revision of a patch to add an option psql to execute 1 or more commands, then exit.

Amit Kapila sent in another revision of a patch to implement parallel seq scan.

Pavel Stěhule and Catalin Iacob traded patches to add a PL/PythonU ereport() function.

David Rowley sent in another revision of a patch to speed up timestamptz_out.

Kyotaro HORIGUCHI sent in two revisions of a patch to fix some tab completion infelicities in psql.

Kaigai Kouhei sent in two more revisions of a patch to add custom scan support to readfuncs.c.

Kyotaro HORIGUCHI sent in two revisions of a patch to fix identifier completion of multibyte characters.

Christian Marie sent in a patch to add length parameterised dmetaphone functions.

Kyotaro HORIGUCHI sent in a patch to allow substitution of ExecScan body for ExecForignScan.

Ashutosh Bapat sent in another revision of a patch to allow getting sorted data from a foreign server for merge joins.

Michael Paquier sent in a patch to add a space-related constraint on the standby snapshot generated by the bgwriter, so as to not rely entirely on the interval of 15s.

Fabien COELHO sent in four more revisions of a patch to extend pgbench expressions with functions.

Ildus Kurbangaliev sent in a patch to split the SLRU LWLocks into separate tranches and move them to SLRU Ctl.

Robert Haas sent in a patch to add a new flag, consider_parallel, to each RelOptInfo, indicating whether a plan for that relation could conceivably be run inside of a parallel worker.

Artur Zakirov sent in two revisions of a patch to improve the use of Hunspell dictionaries.

Vitaly Burovoy sent in a patch to ensure that parts extracted from "infinity" timestamps are also infinite.

Lars Kanis sent in a doc patch to lowercase pqsslAttribute in SGML-id for consistency.

Noah Misch sent in a patch which removes the SlruScanDirCbFindEarliest() test from TruncateMultiXact(), which in turn makes multixact control decisions independent of whether TruncateMultiXact() is successful at unlinking segments.

par N Bougain le jeudi 12 novembre 2015 à 22h16

Publication de PostgreSQL 9.5 Beta 2

Le PostgreSQL Global Development Group annonce aujourd'hui la deuxième bêta de la version 9.5 de PostgreSQL.


Cette version contient toutes les fonctionnalités qui seront dans la version finale. Peu de modifications devraient intervenir.
Les utilisateurs peuvent désormais tester leurs applications avec cette version en préparation de la version finale.

Différences avec la Bêta 1

Les nombreux bogues remontés par les utilisateurs et les contributeurs ont été corrigés. Citons :
  • de nombreux ajustements de la documentation ;
  • le renommage de PQsslAttributes() en PQsslAttributeNames() ;
  • le passage de données de contexte aux pgworkers ;
  • la correction de problèmes avec les workers parallèles ;
  • la correction de divers bogues sur les index BRIN ;
  • la correction de différents problèmes de traçage des timestamp de commit lors de la réplication ;
  • ssl_renegotiation_limit est désormais positionné à 0.


Si vous avez fait un rapport de bogue lors de tests de PostgreSQL 9.5, nous vous demandons de retester avec la Bêta 1 pour vérifier que le problème a été corrigé.

Si vous n'avez pas encore testé PostgreSQL 9.5, c'est le bon moment pour aider au développement de PostgreSQL.
Les problèmes connus et en attente de correction sont listés sur la page des problèmes à régler.

Agenda

Il s'agit de la deuxième version bêta pour la 9.5.
Cela signifie que peu de modifications sont prévues d'ici la version finale.

Le projet PostgreSQL publiera d'autres bêta, si nécessaire, puis une ou plusieurs release candidates, jusqu'à la version finale, fin 2015.

Pour plus d'informations et des suggestions sur la manière de tester les bêta, on peut se référer à la page Beta Testing.

La documentation complète et les notes de version sont disponibles en ligne, et installées avec PostgreSQL. La page What's New donne les détails de chaque nouvelle fonctionnalité.

Liens

par SAS le jeudi 12 novembre 2015 à 22h07

mercredi 4 novembre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 1er novembre 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151101233927.GA7054@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Ãlvaro Herrera pushed:

Robert Haas pushed:

Peter Eisentraut pushed:

Tom Lane pushed:

  • Docs: add example clarifying use of nested JSON containment. Show how this can be used in practice to make queries simpler and more flexible. Also, draw an explicit contrast to the existence operator, which doesn't work that way. Peter Geoghegan and Tom Lane http://git.postgresql.org/pg/commitdiff/23937a4253490bf0c06aef0b8658270176f52de4
  • Implement lookbehind constraints in our regular-expression engine. A lookbehind constraint is like a lookahead constraint in that it consumes no text; but it checks for existence (or nonexistence) of a match *ending* at the current point in the string, rather than one *starting* at the current point. This is a long-requested feature since it exists in many other regex libraries, but Henry Spencer had never got around to implementing it in the code we use. Just making it work is actually pretty trivial; but naive copying of the logic for lookahead constraints leads to code that often spends O(N^2) time to scan an N-character string, because we have to run the match engine from string start to the current probe point each time the constraint is checked. In typical use-cases a lookbehind constraint will be written at the start of the regex and hence will need to be checked at every character --- so O(N^2) work overall. To fix that, I introduced a third copy of the core DFA matching loop, paralleling the existing longest() and shortest() loops. This version, matchuntil(), can suspend and resume matching given a couple of pointers' worth of storage space. So we need only run it across the string once, stopping at each interesting probe point and then resuming to advance to the next one. I also put in an optimization that simplifies one-character lookahead and lookbehind constraints, such as "(?=x)" or "(?<!\w)", into AHEAD and BEHIND constraints, which already existed in the engine. This avoids the overhead of the LACON machinery entirely for these rather common cases. The net result is that lookbehind constraints run a factor of three or so slower than Perl's for multi-character constraints, but faster than Perl's for one-character constraints ... and they work fine for variable-length constraints, which Perl gives up on entirely. So that's not bad from a competitive perspective, and there's room for further optimization if anyone cares. (In reality, raw scan rate across a large input string is probably not that big a deal for Postgres usage anyway; so I'm happy if it's linear.) http://git.postgresql.org/pg/commitdiff/12c9a04008870c283931d6b3b648ee21bbc2cfda

Kevin Grittner pushed:

  • Fix serialization anomalies due to race conditions on INSERT. On insert the CheckForSerializableConflictIn() test was performed before the page(s) which were going to be modified had been locked (with an exclusive buffer content lock). If another process acquired a relation SIReadLock on the heap and scanned to a page on which an insert was going to occur before the page was so locked, a rw-conflict would be missed, which could allow a serialization anomaly to be missed. The window between the check and the page lock was small, so the bug was generally not noticed unless there was high concurrency with multiple processes inserting into the same table. This was reported by Peter Bailis as bug #11732, by Sean Chittenden as bug #13667, and by others. The race condition was eliminated in heap_insert() by moving the check down below the acquisition of the buffer lock, which had been the very next statement. Because of the loop locking and unlocking multiple buffers in heap_multi_insert() a check was added after all inserts were completed. The check before the start of the inserts was left because it might avoid a large amount of work to detect a serialization anomaly before performing the all of the inserts and the related WAL logging. While investigating this bug, other SSI bugs which were even harder to hit in practice were noticed and fixed, an unnecessary check (covered by another check, so redundant) was removed from heap_update(), and comments were improved. Back-patch to all supported branches. Kevin Grittner and Thomas Munro http://git.postgresql.org/pg/commitdiff/585e2a3b1a57cd7f763a9203e77563c729d6104e

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Zeus Kronion sent in a patch to fix an issue with parallel workers in pg_dump.

Haribabu Kommi sent in another revision of a patch to implement multi-tenancy with RLS.

Victor Wagner sent in another revision of a patch to implement failover at the libpq connect level.

Ashutosh Bapat sent in three more revisions of a patch to get sorted data from a foreign server.

Nathan Wagner and David Fetter sent in two revisions of a patch to add fortnights as a unit of time measurement.

Robert Haas sent in a patch to modify tqueue infrastructure to support transient record types.

Valery Popov sent in a patch to add a max_recursion_depth parameter to blunt the effects of out-of-control WITH RECURSIVE queries.

Michael Paquier sent in another revision of a patch to allow showing tuple data in the pageinspect contrib extension.

David Rowley sent in a patch to make tlist_matches_tupdesc() more efficient.

Marko Tiikkaja sent in a patch to add \pset true/false to psql.

Marko Tiikkaja sent in a patch to add a new aggregate, onlyvalue, which returns the single distinct non-NULL value from the input values and raises an exception if some number other than one distinct non-NULL value exists.

Rahila Syed sent in another revision of a patch to implement a vacuum progress checker.

Pavel Stěhule sent in another revision of a patch to implement ereport() in PL/PythonU.

Stas Kelvich sent in another revision of a patch to add KNN support to the cube contrib extension.

Alexander Korotkov sent in two more revisions of a patch to move PinBuffer and UnpinBuffer to atomics.

SAWADA Masahiko sent in two more revisions of a patch to add a "frozen" bit to the visibility map.

Amit Langote sent in another revision of a patch to implement declarative partitioning.

Pavel Stěhule sent in another revision of a patch to enhance the types usable in PL/pgsql.

Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.

Jim Nasby sent in a patch to install config/missing.

Konstantin Knizhnik sent in a patch to add an extensible transaction manager API.

Marko Tiikkaja sent in a patch to allow COPY (query) to use INSERT/UPDATE/DELETE...RETURNING in addition to SELECT, TABLE, and VALUES.

Pavel Stěhule sent in another revision of a patch to add an optional --group-command parameter to psql, which lets it execute multiple commands from the shell command line.

Peter Geoghegan sent in a patch to correct a comment in strxfrm cache.

Alexander Lebedev sent in a patch to add support for box type in SP-GiST index.

Dean Rasheed sent in a patch to add trigonometric functions that take degree measurements as inputs.

par N Bougain le mercredi 4 novembre 2015 à 07h56

mercredi 28 octobre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 25 octobre 2015

Postgres Conference China 2015 aura lieu du 20 au 22 novembre 2015 à Beijing : http://postgres2015.eventdove.com/ http://postgrescluster2015.eventdove.com/

Le Brisbane PostgreSQL Users Group a le plaisir d'annoncer leur première réunion le 27 octobre 2015 à 18h au 4e étage du Fujitsu Building, 1 Breakfast Creek Road, Brisbane, 4006 Queensland. RSVP ci-après : http://www.meetup.com/Brisbane-PostgreSQL-User-Group/events/224709832/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). Le programme a été publié : http://2015.pgconf.eu/
  • Highload++2015 aura lieu les 2 et 3 novembre 2015 à Moscou (Russie). Informations respectivement en russe et anglais : http://highload.ru/ http://highload.co
  • PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre : http://www.pgconfsv.com
  • PgCUBA du 19 au 23 octobre 2015 à la Havane. Informations en espagnol ci-après : http://www.postgresql.org/about/event/1813/
  • PGBR2015 (la PgConf brésilienne) aura lieu à Porto Alegre (État du Rio Grande do Sul) les 18, 19 et 20 novembre. Les inscriptions sont ouvertes : http://pgbr.postgresql.org.br/2015/en/
  • La PGConf.DE se tiendra au Lindner Hotel am Michel de Hambourg (Allemagne) les 26 & 27 novembre 2015 : http://2015.pgconf.de/
  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. L'appel à conférenciers expire le 30 octobre 2015 : https://www.socallinuxexpo.org/scale/14x/cfp

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151026045300.GA24026@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Fix incorrect handling of lookahead constraints in pg_regprefix(). pg_regprefix was doing nothing with lookahead constraints, which would be fine if it were the right kind of nothing, but it isn't: we have to terminate our search for a fixed prefix, not just pretend the LACON arc isn't there. Otherwise, if the current state has both a LACON outarc and a single plain-color outarc, we'd falsely conclude that the color represents an addition to the fixed prefix, and generate an extracted index condition that restricts the indexscan too much. (See added regression test case.) Terminating the search is conservative: we could traverse the LACON arc (thus assuming that the constraint can be satisfied at runtime) and then examine the outarcs of the linked-to state. But that would be a lot more work than it seems worth, because writing a LACON followed by a single plain character is a pretty silly thing to do. This makes a difference only in rather contrived cases, but it's a bug, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/9f1e642d50b30ec5c9604b6095073012a9a73e93
  • Fix incorrect translation of minus-infinity datetimes for json/jsonb. Commit bda76c1c8cfb1d11751ba6be88f0242850481733 caused both plus and minus infinity to be rendered as "infinity", which is not only wrong but inconsistent with the pre-9.4 behavior of to_json(). Fix that by duplicating the coding in date_out/timestamp_out/timestamptz_out more closely. Per bug #13687 from Stepan Perlov. Back-patch to 9.4, like the previous commit. In passing, also re-pgindent json.c, since it had gotten a bit messed up by recent patches (and I was already annoyed by indentation-related problems in back-patching this fix ...) http://git.postgresql.org/pg/commitdiff/d4355425831fe8f6a68095393e3628cb30d06b3f
  • Remove redundant CREATEUSER/NOCREATEUSER options in CREATE ROLE et al. Once upon a time we did not have a separate CREATEROLE privilege, and CREATEUSER effectively meant SUPERUSER. When we invented CREATEROLE (in 8.1) we also added SUPERUSER so as to have a less confusing keyword for this role property. However, we left CREATEUSER in place as a deprecated synonym for SUPERUSER, because of backwards-compatibility concerns. It's still there and is still confusing people, as for example in bug #13694 from Justin Catterson. 9.6 will be ten years or so later, which surely ought to be long enough to end the deprecation and just remove these old keywords. Hence, do so. http://git.postgresql.org/pg/commitdiff/d371bebd3d16949171282c8252dfd6f82d2e7378

Noah Misch pushed:

  • Eschew "RESET statement_timeout" in tests. Instead, use transaction abort. Given an unlucky bout of latency, the timeout would cancel the RESET itself. Buildfarm members gharial, lapwing, mereswine, shearwater, and sungazer witness that. Back-patch to 9.1 (all supported versions). The query_canceled test still could timeout before entering its subtransaction; for whatever reason, that has yet to happen on the buildfarm. http://git.postgresql.org/pg/commitdiff/8e3b4d9d40244c037bbc6e182ea3fabb9347d482

Robert Haas pushed:

Peter Eisentraut pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Thomas Munro sent in three more revisions of a patch to macro-ize psql's tab-complete.c.

YUriy Zhuravlev sent in a patch to allow leaving the ends off an array slice, so foo[2:] would mean the slice of the 1-D foo array going from index 2 to the end.

Craig Ringer sent in another revision of a patch to send replication origin on commit in logical replication.

SAWADA Masahiko sent in a patch to support multi sync replication.

Etsuro Fujita sent in another revision of a patch to fix some infelicities between foreign join pushdown and EvalPlanQual.

Michael Paquier sent in another revision of a patch to ensure that overflow checks not be optimized away.

Alexander Korotkov sent in another revision of a patch to implement partial sorting.

Laurenz Albe sent in a patch to move documentation for min_wal_size before max_wal_size.

Álvaro Herrera sent in another revision of a patch to add a variable to shmem which is used for module activation tracking.

Haribabu Kommi sent in another revision of a patch to implement multi-tenancy with RLS.

Rahila Syed sent in another revision of a patch to implement a VACUUM progress checker.

Rajeev Rastogi and Robert Haas traded patches to fix an issue which caused dangling backend processes.

SAWADA Masahiko sent in two more revisions of a patch to add a "frozen" bit to the visibility map.

Amit Kapila sent in two more revisions of a patch to implement parallel sequential scans.

Taiki Kondo sent in another revision of a patch to consider table partitions when doing join pushdown.

Euler Taveira de Oliveira sent in a patch to fix a duplicate assignment in walsender code.

Euler Taveira de Oliveira sent in a patch to fix some errors in pg_recvlogical.

Ashutosh Bapat sent in another revision of a patch to implement getting sorted data from a foreign server.

Jeff Janes sent in another revision of a patch to make COPY FREEZE and PD_ALL_VISIBLE work together.

Victor Wagner sent in another revision of a patch to implement failover on the libpq connect level.

Dinesh Kumar sent in another revision of a patch to add a pg_report_log() function callable from SQL.

Robert Haas sent in a patch to strengthen planner infrastructure for parallelism.

Robert Haas sent in a patch to fix problems with ParamListInfo serialization mechanism.

Robert Haas sent in a patch set to modify the tqueue infrastructure to support transient record types, and using same, make Gather node projection-capable.

Michael Paquier sent in another revision of a patch to add = and <> operators for xid vs. int.

Jeff Janes sent in a patch to fix psql wrapped format expanded output.

Sandro Santilli sent in another revision of a patch to add an extensions_path GUC.

Etsuro Fujita and Amit Langote each sent in patches to add an Oxford comma to the same error message.

Ian Lawrence Barwick sent in another revision of a patch to improve BRIN documentation for inclusion opclass.

par N Bougain le mercredi 28 octobre 2015 à 08h06

vendredi 23 octobre 2015

Damien Clochard

Assemblée Générale de l'association PostgreSQLFr

L’assemblée générale de l’association PostgreSQLFr se tiendra le mercredi 18 novembre 2015 à 20h30 dans les locaux de la société DALIBO au 10 rue d’Uzès 75018 Paris.

Cette réunion se tiendra en marge du salon Paris Open Source Summit. Si vous passez sur le salon, n’hésitez pas à venir sur le stand de la communauté PostgreSQL et à nous rejoindre pour l’assemblée le soir.

L’ordre du jour provisoire est disponible ici: http://wiki.postgresql.fr/asso:ordre_du_jour_ag_2015

Merci de confirmer votre présence en envoyant un message à l’adresse bureau@postgresql.fr avant le 17 novembre.

par Damien Clochard le vendredi 23 octobre 2015 à 20h52

mercredi 21 octobre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 18 octobre 2015

Highload++2015 aura lieu les 2 et 3 novembre 2015 à Moscou (Russie). Informations respectivement en russe et anglais : http://highload.ru/ http://highload.co

Les nouveautés des produits dérivés

  • pgFormatter v1.5, un outil de formatage/coloration syntaxique qui gère les mots-clefs de SQL-92, SQL-99, SQL-2003, SQL-2008, SQL-2011 et ceux de PostgreSQL qui ne sont pas standards : http://sqlformat.darold.net/
  • pitrery 1.9, a set of bash scripts to manage PITR backups for PostgreSQL, released. http://dalibo.github.io/pitrery/

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

  • Le PGDay.IT 2015 aura lieu à Prato le 23 octobre 2015 : http://pgday.it
  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). Le programme a été publié : http://2015.pgconf.eu/
  • PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre : http://www.pgconfsv.com
  • PgCUBA du 19 au 23 octobre 2015 à la Havane. Informations en espagnol ci-après : http://www.postgresql.org/about/event/1813/
  • PGBR2015 (la PgConf brésilienne) aura lieu à Porto Alegre (État du Rio Grande do Sul) les 18, 19 et 20 novembre. Les inscriptions sont ouvertes : http://pgbr.postgresql.org.br/2015/en/
  • La PGConf.DE se tiendra au Lindner Hotel am Michel de Hambourg (Allemagne) les 26 & 27 novembre 2015 : http://2015.pgconf.de/
  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. L'appel à conférenciers expire le 30 octobre 2015 : https://www.socallinuxexpo.org/scale/14x/cfp

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151019032551.GA20974@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Noah Misch pushed:

Tom Lane pushed:

  • Fix "pg_ctl start -w" to test child process status directly. pg_ctl start with -w previously relied on a heuristic that the postmaster would surely always manage to create postmaster.pid within five seconds. Unfortunately, that fails much more often than we would like on some of the slower, more heavily loaded buildfarm members. We have known for quite some time that we could remove the need for that heuristic on Unix by using fork/exec instead of system() to launch the postmaster. This allows us to know the exact PID of the postmaster, which allows near-certain verification that the postmaster.pid file is the one we want and not a leftover, and it also lets us use waitpid() to detect reliably whether the child postmaster has exited or not. What was blocking this change was not wanting to rewrite the Windows version of start_postmaster() to avoid use of CMD.EXE. That's doable in theory but would require fooling about with stdout/stderr redirection, and getting the handling of quote-containing postmaster switches to stay the same might be rather ticklish. However, we realized that we don't have to do that to fix the problem, because we can test whether the shell process has exited as a proxy for whether the postmaster is still alive. That doesn't allow an exact check of the PID in postmaster.pid, but we're no worse off than before in that respect; and we do get to get rid of the heuristic about how long the postmaster might take to create postmaster.pid. On Unix, this change means that a second "pg_ctl start -w" immediately after another such command will now reliably fail, whereas previously it would succeed if done within two seconds of the earlier command. Since that's a saner behavior anyway, it's fine. On Windows, the case can still succeed within the same time window, since pg_ctl can't tell that the earlier postmaster's postmaster.pid isn't the pidfile it is looking for. To ensure stable test results on Windows, we can insert a short sleep into the test script for pg_ctl, ensuring that the existing pidfile looks stale. This hack can be removed if we ever do rewrite start_postmaster(), but that no longer seems like a high-priority thing to do. Back-patch to all supported versions, both because the current behavior is buggy and because we must do that if we want the buildfarm failures to go away. Tom Lane and Michael Paquier http://git.postgresql.org/pg/commitdiff/6bcce25801c3fcb219e0d92198889ec88c74e2ff
  • On Windows, ensure shared memory handle gets closed if not being used. Postmaster child processes that aren't supposed to be attached to shared memory were not bothering to close the shared memory mapping handle they inherit from the postmaster process. That's mostly harmless, since the handle vanishes anyway when the child process exits -- but the syslogger process, if used, doesn't get killed and restarted during recovery from a backend crash. That meant that Windows doesn't see the shared memory mapping as becoming free, so it doesn't delete it and the postmaster is unable to create a new one, resulting in failure to recover from crashes whenever logging_collector is turned on. Per report from Dmitry Vasilyev. It's a bit astonishing that we'd not figured this out long ago, since it's been broken from the very beginnings of out native Windows support; probably some previously-unexplained trouble reports trace to this. A secondary problem is that on Cygwin (perhaps only in older versions?), exec() may not detach from the shared memory segment after all, in which case these child processes did remain attached to shared memory, posing the risk of an unexpected shared memory clobber if they went off the rails somehow. That may be a long-gone bug, but we can deal with it now if it's still live, by detaching within the infrastructure introduced here to deal with closing the handle. Back-patch to all supported branches. Tom Lane and Amit Kapila http://git.postgresql.org/pg/commitdiff/869f693a36556e9872c4369820ce8289451619f0
  • Fix NULL handling in datum_to_jsonb(). The function failed to adhere to its specification that the "tcategory" argument should not be examined when the input value is NULL. This resulted in a crash in some cases. Per bug #13680 from Boyko Yordanov. In passing, re-pgindent some recent changes in jsonb.c, and fix a rather ungrammatical comment. Diagnosis and patch by Michael Paquier, cosmetic changes by me http://git.postgresql.org/pg/commitdiff/3587cbc34fe87e1bde6c202daba791b68e44cb3a
  • Improve memory-usage accounting in regular-expression compiler. This code previously counted the number of NFA states it created, and complained if a limit was exceeded, so as to prevent bizarre regex patterns from consuming unreasonable time or memory. That's fine as far as it went, but the code paid no attention to how many arcs linked those states. Since regexes can be contrived that have O(N) states but will need O(N^2) arcs after fixempties() processing, it was still possible to blow out memory, and take a long time doing it too. To fix, modify the bookkeeping to count space used by both states and arcs. I did not bother with including the "color map" in the accounting; it can only grow to a few megabytes, which is not a lot in comparison to what we're allowing for states+arcs (about 150MB on 64-bit machines or half that on 32-bit machines). Looking at some of the larger real-world regexes captured in the Tcl regression test suite suggests that the most that is likely to be needed for regexes found in the wild is under 10MB, so I believe that the current limit has enough headroom to make it okay to keep it as a hard-wired limit. In connection with this, redefine REG_ETOOBIG as meaning "regular expression is too complex"; the previous wording of "nfa has too many states" was already somewhat inapropos because of the error code's use for stack depth overrun, and it was not very user-friendly either. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/538b3b8b359fa77cb7e1507113efb788b4e159c9
  • Fix regular-expression compiler to handle loops of constraint arcs. It's possible to construct regular expressions that contain loops of constraint arcs (that is, ^ $ AHEAD BEHIND or LACON arcs). There's no use in fully traversing such a loop at execution, since you'd just end up in the same NFA state without having consumed any input. Worse, such a loop leads to infinite looping in the pullback/pushfwd stage of compilation, because we keep pushing or pulling the same constraints around the loop in a vain attempt to move them to the pre or post state. Such looping was previously recognized in CVE-2007-4772; but the fix only handled the case of trivial single-state loops (that is, a constraint arc leading back to its source state) ... and not only that, it was incorrect even for that case, because it broke the admittedly-not-very-clearly-stated API contract of the pull() and push() subroutines. The first two regression test cases added by this commit exhibit patterns that result in assertion failures because of that (though there seem to be no ill effects in non-assert builds). The other new test cases exhibit multi-state constraint loops; in an unpatched build they will run until the NFA state-count limit is exceeded. To fix, remove the code added for CVE-2007-4772, and instead create a general-purpose constraint-loop-breaking phase of regex compilation that executes before we do pullback/pushfwd. Since we never need to traverse a constraint loop fully, we can just break the loop at any chosen spot, if we add clone states that can replicate any sequence of arc transitions that would've traversed just part of the loop. Also add some commentary clarifying why we have to have all these machinations in the first place. This class of problems has been known for some time --- we had a report from Marc Mamin about two years ago, for example, and there are related complaints in the Tcl bug tracker. I had discussed a fix of this kind off-list with Henry Spencer, but didn't get around to doing something about it until the issue was rediscovered by Greg Stark recently. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/48789c5d23a7f382e3cb721547d5e0af7aaa4585
  • Improve performance of pullback/pushfwd in regular-expression compiler. The previous coding would create a new intermediate state every time it wanted to interchange the ordering of two constraint arcs. Certain regex features such as \Y can generate large numbers of parallel constraint arcs, and if we needed to reorder the results of that, we created unreasonable numbers of intermediate states. To improve matters, keep a list of already-created intermediate states associated with the state currently being considered by the outer loop; we can re-use such states to place all the new arcs leading to the same destination or source. I also took the trouble to redefine push() and pull() to have a less risky API: they no longer delete any state or arc that the caller might possibly have a pointer to, except for the specifically-passed constraint arc. This reduces the risk of re-introducing the same type of error seen in the failed patch for CVE-2007-4772. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/6a7153661d66a00a03ff117c24fa49480b0699c8
  • Fix O(N^2) performance problems in regular-expression compiler. Change the singly-linked in-arc and out-arc lists to be doubly-linked, so that arc deletion is constant time rather than having worst-case time proportional to the number of other arcs on the connected states. Modify the bulk arc transfer operations copyins(), copyouts(), moveins(), moveouts() so that they use a sort-and-merge algorithm whenever there's more than a small number of arcs to be copied or moved. The previous method is O(N^2) in the number of arcs involved, because it performs duplicate checking independently for each copied arc. The new method may change the ordering of existing arcs for the destination state, but nothing really cares about that. Provide another bulk arc copying method mergeins(), which is unused as of this commit but is needed for the next one. It basically is like copyins(), but the source arcs might not all come from the same state. Replace the O(N^2) bubble-sort algorithm used in carcsort() with a qsort() call. These changes greatly improve the performance of regex compilation for large or complex regexes, at the cost of extra space for arc storage during compilation. The original tradeoff was probably fine when it was made, but now we care more about speed and less about memory consumption. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/579840ca05e73ac23a3c2616f050333ba853fb34
  • Improve performance of fixempties() pass in regular-expression compiler. The previous coding took something like O(N^4) time to fully process a chain of N EMPTY arcs. We can't really do much better than O(N^2) because we have to insert about that many arcs, but we can do lots better than what's there now. The win comes partly from using mergeins() to amortize de-duplication of arcs across multiple source states, and partly from exploiting knowledge of the ordering of arcs for each state to avoid looking at arcs we don't need to consider during the scan. We do have to be a bit careful of the possible reordering of arcs introduced by the sort-merge coding of the previous commit, but that's not hard to deal with. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/f5b7d103bc4a97a64f9e8ca83192a96767d9a34c
  • Miscellaneous cleanup of regular-expression compiler. Revert our previous addition of "all" flags to copyins() and copyouts(); they're no longer needed, and were never anything but an unsightly hack. Improve a couple of infelicities in the REG_DEBUG code for dumping the NFA data structure, including adding code to count the total number of states and arcs. Add a couple of missed error checks. Add some more documentation in the README file, and some regression tests illustrating cases that exceeded the state-count limit and/or took unreasonable amounts of time before this set of patches. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/afdfcd3f7617c9b7be5966d66ddabdc2e92eb99b

Robert Haas pushed:

  • Improve INSERT .. ON CONFLICT error message. Peter Geoghegan, reviewed by me. http://git.postgresql.org/pg/commitdiff/b8dd19af50950d4933da7ad60f93388e9d0eb851
  • Have dtrace depend on object files directly, not objfiles.txt Per Mark Johnston, this resolves a build error on FreeBSD related to the fact that dtrace is modifying the generated object files under the hood. Consequently, without this, dtrace gets reinvoked at install time because the object files have been updated. This is a pretty hacky fix, but it shouldn't hurt anything, and it's not clear that it's worth expending any more effort for a feature that not too many people are using. Patch by Mark Johnston. This is arguably back-patchable as a bug fix to the build system, but I'm not certain enough of the consequences to try that. Let's see what the buildfarm (and our packagers) think of this change on master first. http://git.postgresql.org/pg/commitdiff/73537828537239923a0f827a92b20502a3efa52d
  • Allow FDWs to push down quals without breaking EvalPlanQual rechecks. This fixes a long-standing bug which was discovered while investigating the interaction between the new join pushdown code and the EvalPlanQual machinery: if a ForeignScan appears on the inner side of a paramaterized nestloop, an EPQ recheck would re-return the original tuple even if it no longer satisfied the pushed-down quals due to changed parameter values. This fix adds a new member to ForeignScan and ForeignScanState and a new argument to make_foreignscan, and requires changes to FDWs which push down quals to populate that new argument with a list of quals they have chosen to push down. Therefore, I'm only back-patching to 9.5, even though the bug is not new in 9.5. Etsuro Fujita, reviewed by me and by Kyotaro Horiguchi. http://git.postgresql.org/pg/commitdiff/5fc4c26db5120bd90348b6ee3101fcddfdf54800
  • Revert "Have dtrace depend on object files directly, not objfiles.txt" This reverts commit 73537828537239923a0f827a92b20502a3efa52d. Per report from Tom Lane, this breaks parallel builds. http://git.postgresql.org/pg/commitdiff/08fbad0afd62690cc82990c0504529ef238ac24d
  • Don't send protocol messages to a shm_mq that no longer exists. Commit 2bd9e412f92bc6a68f3e8bcb18e04955cc35001d introduced a mechanism for relaying protocol messages from a background worker to another backend via a shm_mq. However, there was no provision for shutting down the communication channel. Therefore, a protocol message sent late in the shutdown sequence, such as a DEBUG message resulting from cranking up log_min_messages, could crash the server. To fix, install an on_dsm_detach callback that disables sending messages to the shm_mq when the associated DSM is detached. http://git.postgresql.org/pg/commitdiff/2ad5c27bb565c26a4b12ea3343331c80f121f269
  • Transfer current command counter ID to parallel workers. Commit 924bcf4f16d54c55310b28f77686608684734f42 correctly forbade parallel workers to modify the command counter while in parallel mode, but it inexplicably neglected to actually transfer the current command counter from leader to workers. This can result in the workers seeing a different set of tuples from the leader, which is bad. Repair. http://git.postgresql.org/pg/commitdiff/423ec0877ff29bc78e48e50362dbcde351a1f025
  • Tighten up application of parallel mode checks. Commit 924bcf4f16d54c55310b28f77686608684734f42 failed to enforce parallel mode checks during the commit of a parallel worker, because we exited parallel mode prior to ending the transaction so that we could pop the active snapshot. Re-establish parallel mode during parallel worker commit. Without this, it's far too easy for unsafe actions during the pre-commit sequence to crash the server instead of hitting the error checks as intended. Just to be extra paranoid, adjust a couple of the sanity checks in xact.c to check not only IsInParallelMode() but also IsParallelWorker(). http://git.postgresql.org/pg/commitdiff/94b4f7e2a635c3027a23b07086f740615b56aa64
  • Invalidate caches after cranking up a parallel worker transaction. Starting a parallel worker transaction changes our notion of which XIDs are in-progress or committed, and our notion of the current command counter ID. Therefore, our view of these caches prior to starting this transaction may no longer valid. Defend against that by clearing them. This fixes a bug in commit 924bcf4f16d54c55310b28f77686608684734f42. http://git.postgresql.org/pg/commitdiff/6de6d96d971c76c8d7bd3b31e6f182906137f913
  • Fix a problem with parallel workers being unable to restore role. check_role() tries to verify that the user has permission to become the requested role, but this is inappropriate in a parallel worker, which needs to exactly recreate the master's authorization settings. So skip the check in that case. This fixes a bug in commit 924bcf4f16d54c55310b28f77686608684734f42. http://git.postgresql.org/pg/commitdiff/82b37765c76b8b2daf6cad3dfb5e5b4a2776d56f
  • Mark more functions parallel-restricted or parallel-unsafe. Commit 7aea8e4f2daa4b39ca9d1309a0c4aadb0f7ed81b was overoptimistic about the degree of safety associated with running various functions in parallel mode. Functions that take a table name or OID as an argument are at least parallel-restricted, because the table might be temporary, and we currently don't allow parallel workers to touch temporary tables. Functions that take a query as an argument are outright unsafe, because the query could be anything, including a parallel-unsafe query. Also, the queue of pending notifications is backend-private, so adding to it from a worker doesn't behave correctly. We could fix this by transferring the worker's queue of pending notifications to the master during worker cleanup, but that seems like more trouble than it's worth for now. In addition to adjusting the pg_proc.h markings, also add an explicit check for this in async.c. http://git.postgresql.org/pg/commitdiff/816e336f12ecabdc834d4cc31bcf966b2dd323dc
  • Rewrite interaction of parallel mode with parallel executor support. In the previous coding, before returning from ExecutorRun, we'd shut down all parallel workers. This was dead wrong if ExecutorRun was called with a non-zero tuple count; it had the effect of truncating the query output. To fix, give ExecutePlan control over whether to enter parallel mode, and have it refuse to do so if the tuple count is non-zero. Rewrite the Gather logic so that it can cope with being called outside parallel mode. Commit 7aea8e4f2daa4b39ca9d1309a0c4aadb0f7ed81b is largely to blame for this problem, though this patch modifies some subsequently-committed code which relied on the guarantees it purported to make. http://git.postgresql.org/pg/commitdiff/bfc78d7196eb28cd4e3d6c24f7e607bacecf1129
  • Prohibit parallel query when the isolation level is serializable. In order for this to be safe, the code which hands true serializability will need to taught that the SIRead locks taken by a parallel worker pertain to the same transaction as those taken by the parallel leader. Some further changes may be needed as well. Until the necessary adaptations are made, don't generate parallel plans in serializable mode, and if a previously-generated parallel plan is used after serializable mode has been activated, run it serially. This fixes a bug in commit 7aea8e4f2daa4b39ca9d1309a0c4aadb0f7ed81b. http://git.postgresql.org/pg/commitdiff/a53c06a13eb4669d08ce8a5199aee51f88dd453e
  • Remove cautions about using volatile from spin.h. Commit 0709b7ee72e4bc71ad07b7120acd117265ab51d0 obsoleted this comment but neglected to update it. Thomas Munro http://git.postgresql.org/pg/commitdiff/78652a3332128c89ae369e03698f7212ca73d022
  • Remove volatile qualifiers from dynahash.c, shmem.c, and sinvaladt.c Prior to commit 0709b7ee72e4bc71ad07b7120acd117265ab51d0, access to variables within a spinlock-protected critical section had to be done through a volatile pointer, but that should no longer be necessary. Thomas Munro http://git.postgresql.org/pg/commitdiff/430008b5a7ac395ab3057377104148e80718045c
  • Remove volatile qualifiers from proc.c and procarray.c Prior to commit 0709b7ee72e4bc71ad07b7120acd117265ab51d0, access to variables within a spinlock-protected critical section had to be done through a volatile pointer, but that should no longer be necessary. Michael Paquier http://git.postgresql.org/pg/commitdiff/d53e3d5fe0b97ebcec5340dc4f44faf2b1a5d52f
  • Allow a parallel context to relaunch workers. This may allow some callers to avoid the overhead involved in tearing down a parallel context and then setting up a new one, which means releasing the DSM and then allocating and populating a new one. I suspect we'll want to revise the Gather node to make use of this new capability, but even if not it may be useful elsewhere and requires very little additional code. http://git.postgresql.org/pg/commitdiff/b0b0d84b3d663a148022e900ebfc164284a95f55
  • Add a C API for parallel heap scans. Using this API, one backend can set up a ParallelHeapScanDesc to which multiple backends can then attach. Each tuple in the relation will be returned to exactly one of the scanning backends. Only forward scans are supported, and rescans must be carefully coordinated. This is not exposed to the planner or executor yet. The original version of this code was written by me. Amit Kapila reviewed it, tested it, and improved it, including adding support for synchronized scans, per review comments from Jeff Davis. Extensive testing of this and related patches was performed by Haribabu Kommi. Final cleanup of this patch by me. http://git.postgresql.org/pg/commitdiff/ee7ca559fcf404f9a3bd99da85c8f4ea9fbc2e92

Bruce Momjian pushed:

Ãlvaro Herrera pushed:

Michael Meskes pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Robert Haas sent in a flock of parallelism-related fixes.

Robert Haas and Amit Kapila traded patches for parallel sequential scan.

Peter Geoghegan sent in a patch to set a sentinel value in the SortSupport code.

kolo hhmow sent in a patch to add an rhost option to PAM auth.

Robbie Harwood sent in another revision of a patch to add GSSAPI encryption support.

Michael Paquier sent in a patch to fix TAP tests on Windows.

Alexander Korotkov sent in another revision of a patch to use pg_rewind when target timeline was switched.

Craig Ringer sent in a patch to fix a typo in replorigin_sesssion_origin.

Victor Wagner sent in another revision of a patch to implement failover on the libpq connect level.

Ashutosh Bapat and Robert Haas traded patches to get sorted data from a foreign server.

Peter Geoghegan sent in two more revisions of a patch to the SortSupport code to correctly distinguish the contents of text buffers.

Kevin Grittner sent in another revision of a patch to implement "snapshot too old," this one configured by time.

Craig Ringer sent in two revisions of a patch to expose origin_id to logical decoding callacks, the lack of which was causing functionality issues.

Dmitry Vasilyev, Amit Kapila, and Kyotaro HORIGUCHI traded patches to fix an issue where some failures on Windows were being caused by too-aggressive error raising.

Michael Paquier sent in another revision of a patch to add some overflow checks in helpful places.

Alexander Korotkov sent in another revision of a patch to do partial sorting.

Jinyu Zhang sent in another revision of a patch to optimize memory allocation in function 'bringetbitmap'.

Pavel Stěhule sent in another revision of a patch to implement DROP DATABASE ... FORCE.

Tom Lane sent in a WIP patch to add lookbehind constraints to the regex engine.

Joe Conway sent in a patch to add a hook for SET ROLE.

Pavel Stěhule sent in two revisions of a patch to add a PL/PythonU wrapper for ereport().

Joe Conway sent in a patch to add a TYPE expression.

Petr Jelinek sent in a patch to add a variable to the shmem which is used for module activation tracking - set to true in ActiveCommitTs() and false in DeactivateCommitTs().

Simon Riggs and Shay Rojansky sent in competing ideas for a way to deal with the fact that ssl_renegotiation limit has been removed as of 9.5.

Jeff Janes sent in a patch to add tab completion for extension versions.

Thomas Munro sent in another revision of a patch to refactor psql's tab-complete.c into a state easier to understand and extend.

par N Bougain le mercredi 21 octobre 2015 à 23h17

Nicolas Thauvin

pitrery 1.10

Un semaine après la sortie de la version 1.9, un collègue découvre un bug dans le strict de restore des fichiers WAL, restore_xlog. Lorsqu'on utilise SSH pour stocker les archives des fichiers WAL, le script ne tient pas compte des paramètres spécifiant l'utilisateur et la machine distante en provenance du fichier de configuration.

Ce qui fait que la restore ne fonctionne pas lorsqu'on démarre PostgreSQL à moins de modifier l'appel à restore_xlog dans le paramètre restore_command du fichier recovery.conf pour y indiquer ces deux informations, car en ligne de commande ils sont bien pris en compte.

On peut aussi ajouter dans le fichier de configuration :

RESTORE_COMMAND="/path/to/restore_xlog -C <config_file> -h <archive_host> -u <archive_user> %f %p"

La version 1.10 corrige ce problème, voir le site de pitrery.

mercredi 21 octobre 2015 à 16h42

Damien Clochard

Installer PostgreSQL 9.5 beta sur debian

La première version beta de PostgreSQL 9.5 est sortie la semaine dernière. Voici un tuto rapide pour installer facilement cette version sur debian

Première étape : Ajouter les dépots “PGDG”, c’est à dire les dépots officiels de la communauté PostgreSQL. Pour cela il suffit de créer un fichier /etc/apt/sources.list.d/pgdg.list et d’y inscrire la ligne ci-dessous :

  deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main 9.5

Le mot clé main permet d’avoir accès à toutes les versions stables de PostgreSQL (de 9.0 à 9.4) et le mot clé 9.5 est nécessaire pour avoir accès à la version beta. J’ai réalisé cette installation sur un serveur Jessie. Evidement si vous êtes sur une autre version debian, remplacez jessie par wheezie ou sid…

Ensuite on récupère la clé du dépot et la liste des paquets :

  sudo apt-get install wget ca-certificates
  wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
  sudo apt-get update && sudo apt-get upgrade

Et voilà !

Il ne reste plus qu’à installer la version 9.5 :

  sudo apt-get install postgresql-9.5

par Damien Clochard le mercredi 21 octobre 2015 à 10h52

mercredi 14 octobre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 11 octobre 2015

PostgreSQL 9.5 Beta 1 disponible : http://www.postgresql.org/about/news/1614/

Les mises à jour de sécurité 9.4.5, 9.3.10, 9.2.14, 9.1.19 et 9.0.23 ont été publiées. Détails ci-après, mettez à jour dès que possible. 9.0.23 est la dernière version de la série 9.0 : http://www.postgresql.org/about/news/1615/
[ndt : article fr. nouvelles versions mineures avec correction de failles de sécurité]

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

  • Le PGDay.IT 2015 aura lieu à Prato le 23 octobre 2015 : http://pgday.it
  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). Le programme a été publié : http://2015.pgconf.eu/
  • PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre : http://www.pgconfsv.com
  • PgCUBA du 19 au 23 octobre 2015 à la Havane. Informations en espagnol ci-après : http://www.postgresql.org/about/event/1813/
  • PGBR2015 (la PgConf brésilienne) aura lieu à Porto Alegre (État du Rio Grande do Sul) les 18, 19 et 20 novembre. Les inscriptions sont ouvertes : http://pgbr.postgresql.org.br/2015/en/
  • La PGConf.DE se tiendra au Lindner Hotel am Michel de Hambourg (Allemagne) les 26 & 27 novembre 2015 : http://2015.pgconf.de/
  • PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. L'appel à conférenciers expire le 30 octobre 2015 : https://www.socallinuxexpo.org/scale/14x/cfp

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151012043827.GA2792@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Stephen Frost pushed:

  • ALTER TABLE .. FORCE ROW LEVEL SECURITY. To allow users to force RLS to always be applied, even for table owners, add ALTER TABLE .. FORCE ROW LEVEL SECURITY. row_security=off overrides FORCE ROW LEVEL SECURITY, to ensure pg_dump output is complete (by default). Also add SECURITY_NOFORCE_RLS context to avoid data corruption when ALTER TABLE .. FORCE ROW SECURITY is being used. The SECURITY_NOFORCE_RLS security context is used only during referential integrity checks and is only considered in check_enable_rls() after we have already checked that the current user is the owner of the relation (which should always be the case during referential integrity checks). Back-patch to 9.5 where RLS was added. http://git.postgresql.org/pg/commitdiff/088c83363a11200f2225f279d4a5c6cc6f9db3d2
  • Do not write out WCOs in Query. The WithCheckOptions list in Query are only populated during rewrite and do not need to be written out or read in as part of a Query structure. Further, move WithCheckOptions to the bottom and add comments to clarify that it is only populated during rewrite. Back-patch to 9.5 with a catversion bump, as we are still in alpha. http://git.postgresql.org/pg/commitdiff/4158cc3793fe0e074dcb58e91103344c8ffcd6df
  • Apply SELECT policies in INSERT/UPDATE+RETURNING. Similar to 7d8db3e, given that INSERT+RETURNING requires SELECT rights on the table, apply the SELECT policies as WCOs to the tuples being inserted. Apply the same logic to UPDATE+RETURNING. Back-patch to 9.5 where RLS was added. http://git.postgresql.org/pg/commitdiff/2ca9d5445c35db8956e4abbf1e653373820e8c0a
  • Add regression tests for INSERT/UPDATE+RETURNING. This adds regressions tests which are specific to INSERT+RETURNING and UPDATE+RETURNING to ensure that the SELECT policies are added as WithCheckOptions (and should therefore throw an error when the policy is violated). Per suggestion from Andres. Back-patch to 9.5 as the prior commit was. http://git.postgresql.org/pg/commitdiff/be400cd25c7f407111b9617dbf6a5fae761754cb
  • Handle append_rel_list in expand_security_qual. During expand_security_quals, we take the security barrier quals on an RTE and create a subquery which evaluates the quals. During this, we have to replace any variables in the outer query which refer to the original RTE with references to the columns from the subquery. We need to also perform that replacement for any Vars in the append_rel_list. Only backpatching to 9.5 as we only go through this process in 9.4 for auto-updatable security barrier views, which UNION ALL queries aren't. Discovered by Haribabu Kommi Patch by Dean Rasheed http://git.postgresql.org/pg/commitdiff/b7aac36245261eba9eb7d18561ce44220b361959

Andres Freund pushed:

  • Re-Align *_freeze_max_age reloption limits with corresponding GUC limits. In 020235a5754 I lowered the autovacuum_*freeze_max_age minimums to allow for easier testing of wraparounds. I did not touch the corresponding per-table limits. While those don't matter for the purpose of wraparound, it seems more consistent to lower them as well. It's noteworthy that the previous reloption lower limit for autovacuum_multixact_freeze_max_age was too high by one magnitude, even before 020235a5754. Discussion: 26377.1443105453@sss.pgh.pa.us Backpatch: back to 9.0 (in parts), like the prior patch http://git.postgresql.org/pg/commitdiff/2596d705bdc4ac886a12720b19ef65b3e67b55f7
  • Remove outdated comment about relation level autovacuum freeze limits. The documentation for the autovacuum_multixact_freeze_max_age and autovacuum_freeze_max_age relation level parameters contained: "Note that while you can set autovacuum_multixact_freeze_max_age very small, or even zero, this is usually unwise since it will force frequent vacuuming." which hasn't been true since these options were made relation options, instead of residing in the pg_autovacuum table (834a6da4f7). Remove the outdated sentence. Even the lowered limits from 2596d70 are high enough that this doesn't warrant calling out the risk in the CREATE TABLE docs. Per discussion with Tom Lane and Alvaro Herrera Discussion: 26377.1443105453@sss.pgh.pa.us Backpatch: 9.0- (in parts) http://git.postgresql.org/pg/commitdiff/10cfd6f8544ea658c3d87821ab8525296363bbdf

Bruce Momjian pushed:

Tom Lane pushed:

  • Docs: explain contrib/pg_stat_statements' handling of GC failure. Failure to perform garbage collection now has a user-visible effect, so explain that and explain that reducing pgss_max is the way to prevent it. Per gripe from Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/f8a5e579d18b268f287d81b8a4bd54955ca90c37
  • Perform an immediate shutdown if the postmaster.pid file is removed. The postmaster now checks every minute or so (worst case, at most two minutes) that postmaster.pid is still there and still contains its own PID. If not, it performs an immediate shutdown, as though it had received SIGQUIT. The original goal behind this change was to ensure that failed buildfarm runs would get fully cleaned up, even if the test scripts had left a postmaster running, which is not an infrequent occurrence. When the buildfarm script removes a test postmaster's $PGDATA directory, its next check on postmaster.pid will fail and cause it to exit. Previously, manual intervention was often needed to get rid of such orphaned postmasters, since they'd block new test postmasters from obtaining the expected socket address. However, by checking postmaster.pid and not something else, we can provide additional robustness: manual removal of postmaster.pid is a frequent DBA mistake, and now we can at least limit the damage that will ensue if a new postmaster is started while the old one is still alive. Back-patch to all supported branches, since we won't get the desired improvement in buildfarm reliability otherwise. http://git.postgresql.org/pg/commitdiff/7e2a18a9161fee7e67642863f72b51d77d3e996f
  • Improve documentation of the role-dropping process. In general one may have to run both REASSIGN OWNED and DROP OWNED to get rid of all the dependencies of a role to be dropped. This was alluded to in the REASSIGN OWNED man page, but not really spelled out in full; and in any case the procedure ought to be documented in a more prominent place than that. Add a section to the "Database Roles" chapter explaining this, and do a bit of wordsmithing in the relevant commands' man pages. http://git.postgresql.org/pg/commitdiff/1ea0c73c2b112911fb2ef297121be92d3b0ca35e
  • Last-minute updates for release notes. Add entries for security and not-quite-security issues. Security: CVE-2015-5288, CVE-2015-5289 http://git.postgresql.org/pg/commitdiff/272ede71a66892e68d4de285a82f117ce23e10ba
  • Fix insufficiently-portable regression test case. Some of the buildfarm members are evidently miserly enough of stack space to pass the originally-committed form of this test. Increase the requirement 10X to hopefully ensure that it fails as-expected everywhere. Security: CVE-2015-5289 http://git.postgresql.org/pg/commitdiff/9e36c91b468d7d821b77214337ff891811b4b337
  • Fix uninitialized-variable bug. For some reason, neither of the compilers I usually use noticed the uninitialized-variable problem I introduced in commit 7e2a18a9161fee7e. That's hardly a good enough excuse though. Committing with brown paper bag on head. In addition to putting the operations in the right order, move the declaration of "now" inside the loop; there's no need for it to be outside, and that does wake up older gcc enough to notice any similar future problem. Back-patch to 9.4; earlier versions lack the time-to-SIGKILL stanza so there's no bug. http://git.postgresql.org/pg/commitdiff/94f5246ce1cac2b3fb976bed5792f39e01a28d08

Noah Misch pushed:

  • Prevent stack overflow in container-type functions. A range type can name another range type as its subtype, and a record type can bear a column of another record type. Consequently, functions like range_cmp() and record_recv() are recursive. Functions at risk include operator family members and referents of pg_type regproc columns. Treat as recursive any such function that looks up and calls the same-purpose function for a record column type or the range subtype. Back-patch to 9.0 (all supported versions). An array type's element type is never itself an array type, so array functions are unaffected. Recursion depth proportional to array dimensionality, found in array_dim_to_jsonb(), is fine thanks to MAXDIM. http://git.postgresql.org/pg/commitdiff/30cb12881de55bc91a2cbde29d836bd3332612c3
  • Prevent stack overflow in query-type functions. The tsquery, ltxtquery and query_int data types have a common ancestor. Having acquired check_stack_depth() calls independently, each was missing at least one call. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/5976097c0fce03f8cc201aefc4445ad57e09bb75
  • Prevent stack overflow in json-related functions. Sufficiently-deep recursion heretofore elicited a SIGSEGV. If an application constructs PostgreSQL json or jsonb values from arbitrary user input, application users could have exploited this to terminate all active database connections. That applies to 9.3, where the json parser adopted recursive descent, and later versions. Only row_to_json() and array_to_json() were at risk in 9.2, both in a non-security capacity. Back-patch to 9.2, where the json type was introduced. Oskari Saarenmaa, reviewed by Michael Paquier. Security: CVE-2015-5289 http://git.postgresql.org/pg/commitdiff/08fa47c4850cea32c3116665975bca219fbf2fe6
  • pgcrypto: Detect and report too-short crypt() salts. Certain short salts crashed the backend or disclosed a few bytes of backend memory. For existing salt-induced error conditions, emit a message saying as much. Back-patch to 9.0 (all supported versions). Josh Kupershmidt Security: CVE-2015-5288 http://git.postgresql.org/pg/commitdiff/1d812c8b059d0b9b1fba4a459c9876de0f6259b6
  • Make prove_installcheck remove the old log directory, if any. prove_check already has been doing this. Back-patch to 9.4, like the commit that introduced this logging. http://git.postgresql.org/pg/commitdiff/03a22f8b1d68754f3ba1fcad977ca44471d21ec1
  • Avoid scan-build warning about uninitialized htonl() arguments. Josh Kupershmidt http://git.postgresql.org/pg/commitdiff/dfa1cddc4c92cce0c35f926d813894a2b3632c27

Robert Haas pushed:

Andrew Dunstan pushed:

Peter Eisentraut pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Heikki Linnakangas sent in a patch to use EVP API pgcrypto encryption, dropping support for OpenSSL 0.9.6 and older.

Stas Kelvich sent in a patch to add tsvector editing functions.

Kyotaro HORIGUCHI sent in two more revisions of a patch to enable index-only scans with partial indexes.

Haribabu Kommi sent in another revision of a patch to implement multi-tenancy with RLS.

SAWADA Masahiko sent in two more revisions of a patch to add a "frozen" bit to the visibility bitmap.

Alexander Korotkov sent in another revision of a patch to refactor the access method interface.

Amit Kapila sent in another revision of a patch to implement parallel seq scan.

Rahila Syed sent in two more revisions of a patch to implement a vacuum progress checker.

Paul Ramsey sent in three more revisions of a patch to add remote extension support to the PostgreSQL FDW.

Nikolay Shaplov sent in a patch to fix a documentation mistake around CREATE TYPE.

Ashutosh Bapat sent in a patch to allow PostgreSQL to request and assume it has gotten sorted data from a foreign server.

David Christensen sent in a patch to teach Catalog.pm how many attributes there should be per DATA() line.

Michael Paquier sent in three more revisions of a patch to add recovery regression tests.

Michael Paquier sent in another revision of a patch to fix pl_ctl start on AIX.

Andreas 'ads' Scherbaum sent in a patch to fix an infelicity in src/bin/initdb/po/zh_CN.po.

Peter Geoghegan sent in another revision of a patch to add SortSupport routine for UUID data type.

Anastasia Lubennikova sent in two revisions of a patch to combine covering and unique functionality for btree indexes.

Pavel Stěhule sent in another revision of a patch to add an ereport function to PL/PythonU.

Andrei Asyakin sent in a patch to process pg_hba.conf keywords as case-insensitive.

Peter Geoghegan sent in a patch to prefetch from memtuples array in tuplesort.

Peter Geoghegan sent in another revision of a patch to reuse abbreviated keys in ordered [set] aggregates.

Peter Geoghegan sent in another revision of a patch to do some minor copy-editing of the INSERT documentation.

par N Bougain le mercredi 14 octobre 2015 à 23h06

jeudi 8 octobre 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 4 octobre 2015

PostgreSQL@SCaLE est un événement de deux jours avec deux programmes qui aura lieu les 21 et 22 janvier 2016 au Pasadena Convention Center à l'occasion du SCaLE 14X. L'appel à conférenciers expire le 30 octobre 2015 : https://www.socallinuxexpo.org/scale/14x/cfp

[ndt 2015-10-09 : nouvelles versions mineures avec correction de failles de sécurité]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20151005003148.GB30093@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Robert Haas pushed:

  • Don't try to create a temp install without abs_top_builddir. Otherwise, we effectively act as if abs_top_builddir were the root directory, which is quite dangerous if the user happens to have permissions to do things there. This can crop up in PGXS builds, for example. Report by Sandro Santilli, patch by me, review by Noah Misch. http://git.postgresql.org/pg/commitdiff/45e5b4ef5c392ddfcd72693f65659bafdda7ee1b
  • Use LOCKBIT_ON() instead of a bit shift in a few places. We do this mostly everywhere, so it seems just as well to do it here, too. Thomas Munro http://git.postgresql.org/pg/commitdiff/f40792a93ccdc4d01c448121da67693fc2f6297a
  • Parallel executor support. This code provides infrastructure for a parallel leader to start up parallel workers to execute subtrees of the plan tree being executed in the master. User-supplied parameters from ParamListInfo are passed down, but PARAM_EXEC parameters are not. Various other constructs, such as initplans, subplans, and CTEs, are also not currently shared. Nevertheless, there's enough here to support a basic implementation of parallel query, and we can lift some of the current restrictions as needed. Amit Kapila and Robert Haas http://git.postgresql.org/pg/commitdiff/d1b7c1ffe72e86932b5395f29e006c3f503bc53d
  • Comment update for join pushdown. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/758fcfdc016ac687c4f4094e0dca6ae641400820
  • Don't dump core when destroying an unused ParallelContext. If a transaction or subtransaction creates a ParallelContext but ends without calling InitializeParallelDSM, the previous code would seg fault. Fix that. http://git.postgresql.org/pg/commitdiff/227d57f3587d7d2a7d0792011f5ac952ba763681
  • Add a Gather executor node. A Gather executor node runs any number of copies of a plan in an equal number of workers and merges all of the results into a single tuple stream. It can also run the plan itself, if the workers are unavailable or haven't started up yet. It is intended to work with the Partial Seq Scan node which will be added in future commits. It could also be used to implement parallel query of a different sort by itself, without help from Partial Seq Scan, if the single_copy mode is used. In that mode, a worker executes the plan, and the parallel leader does not, merely collecting the worker's results. So, a Gather node could be inserted into a plan to split the execution of that plan across two processes. Nested Gather nodes aren't currently supported, but we might want to add support for that in the future. There's nothing in the planner to actually generate Gather nodes yet, so it's not quite time to break out the champagne. But we're getting close. Amit Kapila. Some designs suggestions were provided by me, and I also reviewed the patch. Single-copy mode, documentation, and other minor changes also by me. http://git.postgresql.org/pg/commitdiff/3bd909b220930f21d6e15833a17947be749e7fde
  • Fix readfuncs/outfuncs problems in last night's Gather patch. KaiGai Kohei, with one correction by me. http://git.postgresql.org/pg/commitdiff/286a3a68dcafa15f28a169daac38659816911940
  • Clarify FDW documentation about ON CONFLICT. Etsuro Fujita, reviewed by Peter Geoghegan http://git.postgresql.org/pg/commitdiff/01bc589a464964475a0718497ebd21af5bc7338e

Andres Freund pushed:

  • Fix ON CONFLICT DO UPDATE for tables with oids. When taking the UPDATE path in an INSERT .. ON CONFLICT .. UPDATE tables with oids were not supported. The tuple generated by the update target list was projected without space for an oid - a simple oversight. Reported-By: Peter Geoghegan Author: Andres Freund Backpatch: 9.5, where ON CONFLICT was introduced http://git.postgresql.org/pg/commitdiff/617db3a2d8ad6abc0d7be623f43edc943a6640a8
  • Fix several bugs related to ON CONFLICT's EXCLUDED pseudo relation. Four related issues: 1) attnos/varnos/resnos for EXCLUDED were out of sync when a column after one dropped in the underlying relation was referenced. 2) References to whole-row variables (i.e. EXCLUDED.*) lead to errors. 3) It was possible to reference system columns in the EXCLUDED pseudo relations, even though they would not have valid contents. 4) References to EXCLUDED were rewritten by the RLS machinery, as EXCLUDED was treated as if it were the underlying relation. To fix the first two issues, generate the excluded targetlist with dropped columns in mind and add an entry for whole row variables. Instead of unconditionally adding a wholerow entry we could pull up the expression if needed, but doing it unconditionally seems simpler. The wholerow entry is only really needed for ruleutils/EXPLAIN support anyway. The remaining two issues are addressed by changing the EXCLUDED RTE to have relkind = composite. That fits with EXCLUDED not actually being a real relation, and allows to treat it differently in the relevant places. scanRTEForColumn now skips looking up system columns when the RTE has a composite relkind; fireRIRrules() already had a corresponding check, thereby preventing RLS expansion on EXCLUDED. Also add tests for these issues, and improve a few comments around excluded handling in setrefs.c. Reported-By: Peter Geoghegan, Geoff Winkless. Author: Andres Freund, Amit Langote, Peter Geoghegan. Discussion: CAEzk6fdzJ3xYQZGbcuYM2rBd2BuDkUksmK=mY9UYYDugg_GgZg@mail.gmail.com, CAM3SWZS+CauzbiCEcg-GdE6K6ycHE_Bz6Ksszy8AoixcMHOmsA@mail.gmail.com. Backpatch: 9.5, where ON CONFLICT was introduced http://git.postgresql.org/pg/commitdiff/ad227837924464eb9a7defbe34e95064a3b1bdc2
  • Improve errhint() about replication slot naming restrictions. The existing hint talked about "may only contain letters", but the actual requirement is more strict: only lower case letters are allowed. Reported-By: Rushabh Lathia. Author: Rushabh Lathia. Discussion: AGPqQf2x50qcwbYOBKzb4x75sO_V3g81ZsA8+Ji9iN5t_khFhQ@mail.gmail.com. Backpatch: 9.4-, where replication slots were added http://git.postgresql.org/pg/commitdiff/920218cbc0a36f18793c20184b7ebb999d28e8c8
  • Add missed CREATE EXTENSION ... CASCADE regression test adjustment. http://git.postgresql.org/pg/commitdiff/23fc0b485d6dfa49ff20fff6c5661be4b12a5bff
  • Add CASCADE support for CREATE EXTENSION. Without CASCADE, if an extension has an unfullfilled dependency on another extension, CREATE EXTENSION ERRORs out with "required extension ... is not installed". That is annoying, especially when that dependency is an implementation detail of the extension, rather than something the extension's user can make sense of. In addition to CASCADE this also includes a small set of regression tests around CREATE EXTENSION. Author: Petr Jelinek, editorialized by Michael Paquier, Andres Freund Reviewed-By: Michael Paquier, Andres Freund, Jeff Janes Discussion: 557E0520.3040800@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/b67aaf21e8ef8273d5179a8832a109153d7adfe1
  • Fix hstore_plpython test when python3 is used. Due to b67aaf21e8ef8 / CREATE EXTENSION ... CASCADE the test output contains the extension name in yet another place. Since that's variable depending on the python version... Add yet another name mangling stanza to regress-python3-mangle.mk. Author: Petr Jelinek http://git.postgresql.org/pg/commitdiff/86b1e6784b19b03606197cf3f69b10f5a653931d

Álvaro Herrera pushed:

Stephen Frost pushed:

  • Ensure a few policies remain for pg_upgrade. To make sure that pg_dump/pg_restore function properly with RLS policies, arrange to have a few of them left around at the end of the regression tests. Back-patch to 9.5 where RLS was added. http://git.postgresql.org/pg/commitdiff/992d702bfafdb4258e65cbfa7353a631460a8389
  • Include policies based on ACLs needed. When considering which policies should be included, rather than look at individual bits of the query (eg: if a RETURNING clause exists, or if a WHERE clause exists which is referencing the table, or if it's a FOR SHARE/UPDATE query), consider any case where we've determined the user needs SELECT rights on the relation while doing an UPDATE or DELETE to be a case where we apply SELECT policies, and any case where we've deteremind that the user needs UPDATE rights on the relation while doing a SELECT to be a case where we apply UPDATE policies. This simplifies the logic and addresses concerns that a user could use UPDATE or DELETE with a WHERE clauses to determine if rows exist, or they could use SELECT .. FOR UPDATE to lock rows which they are not actually allowed to modify through UPDATE policies. Use list_append_unique() to avoid adding the same quals multiple times, as, on balance, the cost of checking when adding the quals will almost always be cheaper than keeping them and doing busywork for each tuple during execution. Back-patch to 9.5 where RLS was added. http://git.postgresql.org/pg/commitdiff/7d8db3e8f37aec9d252353904e77381a18a2fa9f

Tom Lane pushed:

  • Fix poor errno handling in libpq's version of our custom OpenSSL BIO. Thom Brown reported that SSL connections didn't seem to work on Windows in 9.5. Asif Naeem figured out that the cause was my_sock_read() looking at "errno" when it needs to look at "SOCK_ERRNO". This mistake was introduced in commit 680513ab79c7e12e402a2aad7921b95a25a4bcc8, which cloned the backend's custom SSL BIO code into libpq, and didn't translate the errno handling properly. Moreover, it introduced unnecessary errno save/restore logic, which was particularly confusing because it was incomplete; and it failed to check for all three of EINTR, EAGAIN, and EWOULDBLOCK in my_sock_write. (That might not be necessary; but since we're copying well-tested backend code that does do that, it seems prudent to copy it faithfully.) http://git.postgresql.org/pg/commitdiff/60f1e6bc130241c1f1da337d0a08a05ee8b982d8
  • Fix plperl to handle non-ASCII error message texts correctly. We were passing error message texts to croak() verbatim, which turns out not to work if the text contains non-ASCII characters; Perl mangles their encoding, as reported in bug #13638 from Michal Leinweber. To fix, convert the text into a UTF8-encoded SV first. It's hard to test this without risking failures in different database encodings; but we can follow the lead of plpython, which is already assuming that no-break space (U+00A0) has an equivalent in all encodings we care about running the regression tests in (cf commit 2dfa15de5). Back-patch to 9.1. The code is quite different in 9.0, and anyway it seems too risky to put something like this into 9.0's final minor release. Alex Hunsaker, with suggestions from Tim Bunce and Tom Lane http://git.postgresql.org/pg/commitdiff/b631a46ed83b7eebf5cde16b41d842596cbcc69d
  • Small improvements in comments in async.c. We seem to have lost a line somewhere along the way in the comment block that discusses async.c's locks, because it suddenly refers to "both locks" without previously having mentioned more than one. Add a sentence to make that read more sanely. Also, refer to the "pos of the slowest backend" not the "tail of the slowest backend", since we have no per-backend value called "tail". http://git.postgresql.org/pg/commitdiff/6057f61b4d511d8321f295d827ced13809e5a1ae
  • Improve LISTEN startup time when there are many unread notifications. If some existing listener is far behind, incoming new listener sessions would start from that session's read pointer and then need to advance over many already-committed notification messages, which they have no interest in. This was expensive in itself and also thrashed the pg_notify SLRU buffers a lot more than necessary. We can improve matters considerably in typical scenarios, without much added cost, by starting from the furthest-ahead read pointer, not the furthest-behind one. We do have to consider only sessions in our own database when doing this, which requires an extra field in the data structure, but that's a pretty small cost. Back-patch to 9.0 where the current LISTEN/NOTIFY logic was introduced. Matt Newell, slightly adjusted by me http://git.postgresql.org/pg/commitdiff/07e4d03fb476587bd943b4ba7d51bf0bb559a631
  • Fix errors in commit a04bb65f70dafdf462e0478ad19e6de56df89bfc. Not a lot of commentary needed here really. http://git.postgresql.org/pg/commitdiff/5884b92a841d83ceadb986843892d57c68ea32cd
  • Fix documentation error in commit 8703059c6b55c427100e00a09f66534b6ccbfaa1. Etsuro Fujita spotted a thinko in the README commentary. http://git.postgresql.org/pg/commitdiff/21995d3f6d69670b5bc5b3654970701f8abb6d9f
  • Fix pg_dump to handle inherited NOT VALID check constraints correctly. This case seems to have been overlooked when unvalidated check constraints were introduced, in 9.2. The code would attempt to dump such constraints over again for each child table, even though adding them to the parent table is sufficient. In 9.2 and 9.3, also fix contrib/pg_upgrade/Makefile so that the "make clean" target fully cleans up after a failed test. This evidently got dealt with at some point in 9.4, but it wasn't back-patched. I ran into it while testing this fix ... Per bug #13656 from Ingmar Brouns. http://git.postgresql.org/pg/commitdiff/8ab4a6bd3f1a300313eab04d52b0c9a4003f3e43
  • Docs: add another example of creating a range type. The "floatrange" example is a bit too simple because float8mi can be used without any additional type conversion. Add an example that does have to account for that, and do some minor other wordsmithing. http://git.postgresql.org/pg/commitdiff/27fddec197857db4d54db50a63f311bb0ed7b6fd
  • Docs: add disclaimer about hazards of using regexps from untrusted sources. It's not terribly hard to devise regular expressions that take large amounts of time and/or memory to process. Recent testing by Greg Stark has also shown that machines with small stack limits can be driven to stack overflow by suitably crafted regexps. While we intend to fix these things as much as possible, it's probably impossible to eliminate slow-execution cases altogether. In any case we don't want to treat such things as security issues. The history of that code should already discourage prudent DBAs from allowing execution of regexp patterns coming from possibly-hostile sources, but it seems like a good idea to warn about the hazard explicitly. Currently, similar_escape() allows access to enough of the underlying regexp behavior that the warning has to apply to SIMILAR TO as well. We might be able to make it safer if we tightened things up to allow only SQL-mandated capabilities in SIMILAR TO; but that would be a subtly non-backwards-compatible change, so it requires discussion and probably could not be back-patched. Per discussion among pgsql-security list. http://git.postgresql.org/pg/commitdiff/558d4ada1851274fe4dd3618f3f6561b63857e8f
  • Add some more query-cancel checks to regular expression matching. Commit 9662143f0c35d64d7042fbeaf879df8f0b54be32 added infrastructure to allow regular-expression operations to be terminated early in the event of SIGINT etc. However, fuzz testing by Greg Stark disclosed that there are still cases where regex compilation could run for a long time without noticing a cancel request. Specifically, the fixempties() phase never adds new states, only new arcs, so it doesn't hit the cancel check I'd put in newstate(). Add one to newarc() as well to cover that. Some experimentation of my own found that regex execution could also run for a long time despite a pending cancel. We'd put a high-level cancel check into cdissect(), but there was none inside the core text-matching routines longest() and shortest(). Ordinarily those inner loops are very very fast ... but in the presence of lookahead constraints, not so much. As a compromise, stick a cancel check into the stateset cache-miss function, which is enough to guarantee a cancel check at least once per lookahead constraint test. Making this work required more attention to error handling throughout the regex executor. Henry Spencer had apparently originally intended longest() and shortest() to be incapable of incurring errors while running, so neither they nor their subroutines had well-defined error reporting behaviors. However, that was already broken by the lookahead constraint feature, since lacon() can surely suffer an out-of-memory failure --- which, in the code as it stood, might never be reported to the user at all, but just silently be treated as a non-match of the lookahead constraint. Normalize all that by inserting explicit error tests as needed. I took the opportunity to add some more comments to the code, too. Back-patch to all supported branches, like the previous patch. http://git.postgresql.org/pg/commitdiff/9fe8fe9c9e5d7fc099acfc96e976ee72b2b49865
  • Fix potential infinite loop in regular expression execution. In cfindloop(), if the initial call to shortest() reports that a zero-length match is possible at the current search start point, but then it is unable to construct any actual match to that, it'll just loop around with the same start point, and thus make no progress. We need to force the start point to be advanced. This is safe because the loop over "begin" points has already tried and failed to match starting at "close", so there is surely no need to try that again. This bug was introduced in commit e2bd904955e2221eddf01110b1f25002de2aaa83, wherein we allowed continued searching after we'd run out of match possibilities, but evidently failed to think hard enough about exactly where we needed to search next. Because of the way this code works, such a match failure is only possible in the presence of backrefs --- otherwise, shortest()'s judgment that a match is possible should always be correct. That probably explains how come the bug has escaped detection for several years. The actual fix is a one-liner, but I took the trouble to add/improve some comments related to the loop logic. After fixing that, the submitted test case "()*\1" didn't loop anymore. But it reported failure, though it seems like it ought to match a zero-length string; both Tcl and Perl think it does. That seems to be from overenthusiastic optimization on my part when I rewrote the iteration match logic in commit 173e29aa5deefd9e71c183583ba37805c8102a72: we can't just "declare victory" for a zero-length match without bothering to set match data for capturing parens inside the iterator node. Per fuzz testing by Greg Stark. The first part of this is a bug in all supported branches, and the second part is a bug since 9.2 where the iteration rewrite happened. http://git.postgresql.org/pg/commitdiff/f2c4ffc3307cab6619a28e77da9211416c8b1d83
  • Add recursion depth protections to regular expression matching. Some of the functions in regex compilation and execution recurse, and therefore could in principle be driven to stack overflow. The Tcl crew has seen this happen in practice in duptraverse(), though their fix was to put in a hard-wired limit on the number of recursive levels, which is not too appetizing --- fortunately, we have enough infrastructure to check the actually available stack. Greg Stark has also seen it in other places while fuzz testing on a machine with limited stack space. Let's put guards in to prevent crashes in all these places. Since the regex code would leak memory if we simply threw elog(ERROR), we have to introduce an API that checks for stack depth without throwing such an error. Fortunately that's not difficult. http://git.postgresql.org/pg/commitdiff/b63fc28776c5d2efdb4de326ad0f0b5b88f82220
  • Add recursion depth protection to LIKE matching. Since MatchText() recurses, it could in principle be driven to stack overflow, although quite a long pattern would be needed. http://git.postgresql.org/pg/commitdiff/2e8cfcf4ea1140213eae56ce64a2c53182643578
  • Update time zone data files to tzdata release 2015g. DST law changes in Cayman Islands, Fiji, Moldova, Morocco, Norfolk Island, North Korea, Turkey, Uruguay. New zone America/Fort_Nelson for Canadian Northern Rockies. http://git.postgresql.org/pg/commitdiff/241e6844ad7643783e9b4bc020d2abe6a9cbfed2
  • Add missing "static" specifier. Per buildfarm (pademelon, at least, doesn't like this). http://git.postgresql.org/pg/commitdiff/bf686796a055a8bd4abc9765763eb675a0428473
  • First-draft release notes for 9.4.5, 9.3.10, 9.2.14, 9.1.19, 9.0.23. http://git.postgresql.org/pg/commitdiff/01ef33701bf6d475deeb550c18a5c3fd698c9623
  • Update 9.5 release notes through today. http://git.postgresql.org/pg/commitdiff/cf007a4bca59a1bca6951351e10ff227d2b0c10a
  • Fix some issues in new hashtable size calculations in nodeHash.c. Limit the size of the hashtable pointer array to not more than MaxAllocSize, per reports from Kouhei Kaigai and others of "invalid memory alloc request size" failures. There was discussion of allowing the array to get larger than that by using the "huge" palloc API, but so far no proof that that is actually a good idea, and at this point in the 9.5 cycle major changes from old behavior don't seem like the way to go. Fix a rather serious secondary bug in the new code, which was that it didn't ensure nbuckets remained a power of 2 when recomputing it for the multiple-batch case. Clean up sloppy division of labor between ExecHashIncreaseNumBuckets and its sole call site. http://git.postgresql.org/pg/commitdiff/a31e64d0652622d4627ffa2123163a43538a3f51
  • Further twiddling of nodeHash.c hashtable sizing calculation. On reflection, the submitted patch didn't really work to prevent the request size from exceeding MaxAllocSize, because of the fact that we'd happily round nbuckets up to the next power of 2 after we'd limited it to max_pointers. The simplest way to enforce the limit correctly is to round max_pointers down to a power of 2 when it isn't one already. (Note that the constraint to INT_MAX / 2, if it were doing anything useful at all, is properly applied after that.) http://git.postgresql.org/pg/commitdiff/f2fc98fb8e060b2243aba4cfe8b206f54bcc5a4f
  • Improve contrib/pg_stat_statements' handling of garbage collection failure. If we can't read the query texts file (whether because out-of-memory, or for some other reason), give up and reset the file to empty, discarding all stored query texts, though not the statistics per se. We used to leave things alone and hope for better luck next time, but the problem is that the file is only going to get bigger and even harder to slurp into memory. Better to do something that will get us out of trouble. Likewise reset the file to empty for any other failure within gc_qtexts(). The previous behavior after a write error was to discard query texts but not do anything to truncate the file, which is just weird. Also, increase the maximum supported file size from MaxAllocSize to MaxAllocHugeSize; this makes it more likely we'll be able to do a garbage collection successfully. Also, fix recalculation of mean_query_len within entry_dealloc() to match the calculation in gc_qtexts(). The previous coding overlooked the possibility of dropped texts (query_len == -1) and would underestimate the mean of the remaining entries in such cases, thus possibly causing excess garbage collection cycles. In passing, add some errdetail to the log entry that complains about insufficient memory to read the query texts file, which after all was Jim Nasby's original complaint. Back-patch to 9.4 where the current handling of query texts was introduced. Peter Geoghegan, rather editorialized upon by me http://git.postgresql.org/pg/commitdiff/8bbe4cbd9bd687eddfb8fd8f7bba488ff794d166
  • Release notes for 9.5beta1, 9.4.5, 9.3.10, 9.2.14, 9.1.19, 9.0.23. http://git.postgresql.org/pg/commitdiff/16a70e3059885739f59ccdaa20f2e4a3b2a0a700

Andrew Dunstan pushed:

Tatsuo Ishii pushed:

  • Fix incorrect tps number calculation in "excluding connections establishing". The tolerance (larger than actual tps number) increases as the number of threads decreases. The bug has been there since the thread support was introduced in 9.0. Because back patching introduces incompatible behavior changes regarding the tps number, the fix is committed to master and 9.5 stable branches only. Problem spotted by me and fix proposed by Fabien COELHO. Note that his original patch included more than fixes (a code re-factoring) which is not related to the problem and I omitted the part. http://git.postgresql.org/pg/commitdiff/a16db3a07da9c47a5ef5b1d18f9c45bdb893b243

Fujii Masao pushed:

Noah Misch pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Amir Rohan sent in another revision of a patch to better document base backups.

David Rowley sent in two more revisions of a patch to allow GROUP BY before JOIN, which can immensely optimize performance of certain queries.

Marti Raudsepp sent in a patch to skip ALTER x SET SCHEMA if the schema didn't change, rather than erroring out.

Fabien COELHO sent in another revision of a patch to correct some TPS numbers in pgbench.

Kaigai Kouhei sent in two more revisions of a patch to fix an infelicity between foreign join pushdown and EvalPlanQual.

Alexander Shulgin sent in another revision of a patch to add an SHM table of contents to the explain DSM.

Fabien COELHO sent in two more revisions of a patch to add more features to pgbench.

Paul Ramsey and Michael Paquier traded patches to add remote extension support to the PostgreSQL FDW.

Michael Paquier sent in another revision of a patch to add a test facility to check dump/restore with extensions.

David Rowley sent in another revision of a patch to use foreign keys to improve join estimates.

Joshua Elsasser sent in a flock of patches to add a single tar output format to pg_basebackup, fixing some bugs along the way.

Adam Brightwell sent in a patch to document a possible information leak inherent in how tuple visibility works.

Michael Paquier sent in another revision of a patch to handle timeline switches in pg_rewind.

Tomas Vondra sent in two more revisions of a patch to allow index-only scans to use partial indexes.

Matt Newell sent in two revisions of a patch to fix a LISTEN denial of service with aborted transaction.

Nikolay Shaplov and Michael Paquier traded patches to add an ability to show tuple data to the pageinspect utility.

Tom Lane sent in two revisions of a patch to make the postmaster die if $PGDATA disappears.

Stephen Frost sent in another revision of a patch to reserve the namespaces starting with pg_ for roles.

Michael Paquier sent in another revision of a patch to remove obsolete use of volatile in WAL-related files.

SAWADA Masahiko sent in two more revisions of a patch to add a "frozen" bit to the visibility map.

Dinesh Kumar sent in a patch to allow setting permissions on the output file COPY produces via the new syntax, COPY ... [[NO] READ ACCESS].

Alexander Korotkov sent in another revision of a patch to rework the access method interface.

Michael Paquier sent in another revision of a patch to add in-core tests for recovery.

Peter Geoghegan sent in two revisions of a patch to fix an issue that manifested as an error reporting issue in pg_stat_statements.

Peter Geoghegan sent in a patch to extend SortSupport for text with strcoll() and strxfrm() caching.

Peter Geoghegan sent in a patch to add SortSupport for the UUID type.

Michael Paquier sent in two more revisions of a patch to add in-core tests for recovery.

Marko Tiikkaja sent in a patch to add a CONTEXT line to index and constraint rebuilds.

par N Bougain le jeudi 8 octobre 2015 à 22h47

mardi 22 septembre 2015

Guillaume Lelarge

Version beta 0.4 du livre

La dernière beta datait de mi-mai. Beaucoup de choses se sont passées pendant les 4 mois qui ont suivi. Quatre nouveaux chapitres sont mis à disposition :

  • Sauvegarde
  • Réplication
  • Statistiques
  • Maintenance

Mais ce n'est évidemment pas tout. Dans les nouveautés importantes, notons :

  • Chapitres Fichiers, Processus et Mémoire
    • Ajout des schémas disques/processus/mémoire
  • Chapitre Contenu physique des fichiers
    • Déplacement des informations sur le contenu des journaux de transactions dans ce chapitre
    • Ajout de la description du contenu d'un index B-tree
    • Ajout de la description du contenu d'un index Hash
    • Ajout de la description du contenu d'un index BRIN
    • Restructuration du chapitre dans son ensemble
  • Chapitre Architecture des processus
    • Ajout de sous-sections dans la description des processus postmaster et startup
    • Ajout d'un exemple sur la mort inattendue d'un processus du serveur PostgreSQL
  • Chapitre Architecture mémoire
    • Ajout de plus de détails sur la mémoire cache (shared_buffers)
  • Chapitre Gestion des transactions
    • Ajout d'informations sur le CLOG, le FrozenXid et les Hint Bits
  • Chapitre Gestion des objets
    • Ajout d'une section sur les options spécifiques des vues et fonctions pour la sécurité
    • Ajout d'un paragraphe sur le pseudo-type serial
  • Divers
    • Mise à jour des exemples avec PostgreSQL 9.4.4

Bref, c'est par ici.

Quant à la prochaine version ? cela devrait être la version finale. Elle comportera le chapitre Sécurité (déjà écrit, en cours de relecture) et le chapitre sur le planificateur de requêtes (en cours d'écriture). Elle devrait aussi disposer d'une mise à jour complète concernant la version 9.5 (dont la beta devrait sortir début octobre).

Bonne lecture et toujours intéressé pour savoir ce que vous en pensez (via la forum mis en place par l'éditrice ou via mon adresse email).

par Guillaume Lelarge le mardi 22 septembre 2015 à 21h59

lundi 10 août 2015

Rodolphe Quiédeville

Utiliser pg_shard avec Django

L'hiver dernier CitusData à ouvert le code source de son outil de partitionnement pg_shard, le code est désormais publié sous licence LGPL version 3, et disponible sur github. Le 30 juillet dernier la version 1.2 a été releasé, ce fut l'occasion pour moi de tester la compatibilité de Django avec cette nouvelle extension PostgreSQL.

Pour rappel le sharding permet de distribuer le contenu d'une table sur plusieurs serveurs, pg_shard permet également de gérer de multiples copies d'un même réplicats afin de palier à une éventulle faille sur l'un des noeuds. L'intérêt principal du sharding est de pouvoir garantir la scalabilité quand le volume de données augmente rapidement, l'accés aux données se faisant toujours sur le noeud principal sans avoir à prendre en compte les noeuds secondaires qui sont trasparents pour le client.

Autant le dire tout de suite, et ne pas laisser le suspens s'installer, Django n'est pas compatible avec pg_shard, cela pour trois raisons principales détaillée ci-dessous. D'auutres points sont peut-être bloquant, mais je n'ai pas introspecté plus en avant après avoir déjà constaté ces premiers points de blocage.

Lors de la sauvegarde d'un nouvel objet dans la base Django utilise la clause RETURNING dans l'INSERT afin de récupérer l'id de l'objet. A ce jour pg_shard ne supporte pas RETURNING, un ticket est en cours, espérons qu'une future version soit publiée avec cette fonctionnalité.

Plus problématique car cela demanderai un hack un peu plus profond dans l'ORM de Django, le non support des séquences qui sont utilisées par le type SERIAL afin de bénéficier de la numérotation automatique et unique des clés primaires. C'est ce type qui est utilisé par défaut par Django pour les pk. Là encore des discussions sont en cours pour supporter les sequences dans pg_shard.

Enfin et c'est peut-être ce qui serait le plus bloquant pour une utilisation avec Django ou un autre ORM, pg_shard ne supporte pas les transactions multi-requêtes. Les transactions étant la base de la garantie de l'intégrité des données ; à part être dans un cas d'usage où l'on ne modifie pas plus d'une donnée à la fois, cela peut être une raison pour ne pas adopter pg_shard dans l'état.

Malgré ces constats pg_shard reste une solution très intéressante, qu'il faut garder dans un coin de sa veille techno, à l'époque où le big data revient si souvent dans les conversations autour de la machine à café.

par Rodolphe Quiédeville le lundi 10 août 2015 à 10h31

mercredi 5 août 2015

Rodolphe Quiédeville

pgBouncer dans un contexte Django

PgBouncer est un gestionnaire de pool de connexion pour PostgreSQL très efficace, il permet de réduire drastiquement le temps de connexion à la base depuis votre client.

Dans un contexte d'utilisation avec Django l'intérêt peut ne pas apparaître de suite, le temps passé dans l'exécution et la récupération de la requête étant souvent bien supérieur au temps de connexion. Ce paradigme tend à s'inverser dans un contexte d'API ; j'ai eu récemment l'occasion de mesurer l'impact de son utilisation sur un cas réel suite à un problème de timeout sur une API.

L'API est consommée à des taux certes raisonnables, autour de 25 appels par secondes, mais l'accroissement régulier faisait apparaitre des TIMEOUT de plus en plus souvent au niveau du client. En frontal les appels sont reçus par Nginx qui renvoit ceux-ci à des process gunicorn, le timeout coté Nginx est de 60 secondes, c'est ce timeout qui se déclenche. Les mesures sur l'infra de tests de performances continus montraient des temps de réponses de l'ordre de 120msec sous faible charge, ce qui n'était pas cohérent avec les 60 sec du timeout de Nginx.

Seulement après une revue complète de l'infrastucture du SI il est apparu que sur l'environnement de test pgbouncer était installé et correctement configuré, alors que cela n'était le cas du coté de la production. J'ai alors mené une série de tests avec et sans pgbouncer sur la même architecture, afin de mesurer son impacte réel ; PgBouncer faisant partie des préconisations initiales que j'avais faite sur ce projet.

Le test effectue un appel simple avec des données aléatoire et injecte un nombre croissant d'utilisateur pour arriver au plateau de 60 users/sec; il a été mené avec Gatling.

Les premiers tests avec pgbouncer donnent des temps de réponses médians de 285ms avec un 99th percentile à 1650ms, toutes les requêtes sont traitées avec succès

with-pgbouncer.png

Si on débranche pgbouncer le temps de réponses médian croit à 14487ms et surtout le max dépasse 60126ms ce qui donne un nombre croissant de requête en timeout sur la fin du test quand on arrive à pleine charge.

without-pgbouncer.png

Sur la plateforme de test PgBouncer est installé sur la machine qui fait tourner les process gunicorn, le configuration de Django est donc positionnée sur le loopback. La base de données PostgreSQL est elle sur une machine distante avec une connexion sur le LAN.

PgBouncer peut apparaître comme un outil compliqué quand on a pas l'habitude des bases de données, mais il est fort à parier que votre DBA le connait déjà, alors si l'utilisation de vos API croît ayez le réflex PgBouncer !

par Rodolphe Quiédeville le mercredi 5 août 2015 à 11h23

mardi 14 juillet 2015

Guillaume Lelarge

Comment quantifier le maintenance_work_mem

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Je suis en train d'écrire le chapitre sur la maintenance. Parmi les opérations de maintenance se trouve l'ordre VACUUM. Beaucoup de choses ont déjà été écrites dans le livre sur le VACUUM mais j'avais bizarrement oublié une chose. Une bonne configuration du paramètre maintenance_work_mem permet d'avoir un VACUUM performant. Mais comment peut-on savoir que la valeur du maintenance_work_mem est suffisante ?

J'ai donc creusé hier soir dans les sources de PostgreSQL à la recherche de ce qui est stocké dans cette mémoire. Tout se trouve dans src/backend/commands/vacuumlazy.c, principalement dans la fonction lazy_space_alloc(). En gros, PostgreSQL y place un tableau de la structure ItemPointerData. Cette structure prend six octets. Donc une estimation (très grosse) serait de dire qu'on peut stocker maintenance_work_mem/6 positions d'enregistrements morts dans cette mémoire. Un patch rapide (voir le fichier joint) nous prouve cette théorie :

Nous plaçons le paramètre client_min_messages au niveau log pour voir les traces ajoutées par le patch :

postgres=# SET client_min_messages TO log;
SET

Nous créons la table et désactivons l'autovacuum sur cette table pour le gérer nous-même :

postgres=# DROP TABLE IF EXISTS t1;
DROP TABLE
postgres=# CREATE TABLE t1(id INTEGER PRIMARY KEY);
CREATE TABLE
postgres=# ALTER TABLE t1 SET (autovacuum_enabled = OFF);
ALTER TABLE

Nous insérons un million de lignes, puis en supprimons 900000 :

postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999

Nous configurons maintenance_work_mem à 1 Mo (en fait, suffisamment petit pour voir que le VACUUM a besoin de plusieurs passes dû au manque de mémoire) :

postgres=# SET maintenance_work_mem TO '1MB';
SET
postgres=# VACUUM t1;
LOG:  patch - vac_work_mem: 1024
LOG:  patch - sizeof(ItemPointerData): 6
LOG:  patch - maxtuples: 174762
LOG:  patch - step 1
LOG:  patch - step 2
LOG:  patch - step 3
LOG:  patch - step 4
LOG:  patch - step 5
LOG:  patch - step 6
VACUUM

La fonction de calcul de la taille mémoire a bien noté le maintenance_work_mem à 1 Mo (1024 Ko). La taille de la structure est bien de 6 octets. Il est donc possible de stocker 1024*1024/6 enregistrements, soit 174762 enregistrements. Ayant supprimé 900000 enregistrements, il me faut 6 passes (l'arrondi supérieur de l'opération 900000/174762) pour traiter la table entière. Pas efficace.

Essayons dans les mêmes conditions mais avec un maintenance_work_mem trois fois plus gros :

postgres=# TRUNCATE t1;
TRUNCATE TABLE
postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999
postgres=# SET maintenance_work_mem TO '3MB';
SET
postgres=# VACUUM t1;
LOG:  patch - vac_work_mem: 3072
LOG:  patch - sizeof(ItemPointerData): 6
LOG:  patch - maxtuples: 524288
LOG:  patch - step 1
LOG:  patch - step 2
VACUUM

Nous ne faisons plus que deux passes (tout d'abord 524288 enregistrements, puis 375711), c'est plus efficace mais non optimal.

Essayons maintenant avec le maintenance_work_mem de base (64 Mo) :

postgres=# TRUNCATE t1;
TRUNCATE TABLE
postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999
postgres=# RESET maintenance_work_mem;
RESET
postgres=# VACUUM VERBOSE t1;
INFO:  vacuuming "public.t1"
LOG:  patch - vac_work_mem: 65536
LOG:  patch - sizeof(ItemPointerData): 6
LOG:  patch - maxtuples: 1287675
LOG:  patch - step 1
VACUUM

Seule une passe est réalisée. Il est à noter que la mémoire prise ne correspond pas au 64 Mo. 64 Mo me permet de stocker 11 millions d'enregistrements morts, mais je n'ai dans la table que 1000000 d'enregistrements dont 90% est mort. Autrement dit, j'ai besoin de beaucoup moins de mémoire. C'est bien le cas ici où, au lieu de 11 millions d'enregistrements, on peut en stocker 1287675 (soit un peu plus de 7 Mo).

De tout ça, comment puis-je savoir si mon maintenance_work_mem est bien configuré ? Il faut se baser sur le nombre d'enregistrements (morts) contenus dans les tables. Ça correspond à cette requête pour les tables de ma base de connexion :

SELECT pg_size_pretty(max(n_dead_tup*6)) AS custom_maintenance_work_mem
FROM pg_stat_all_tables;

Dans l'exemple précédent, cela me donnerait ceci :

postgres=# TRUNCATE t1;
TRUNCATE TABLE
postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999
postgres=# SELECT pg_size_pretty(max(n_dead_tup*6)) AS custom_maintenance_work_mem,
           current_setting('maintenance_work_mem') AS current_maintenance_work_mem
          FROM pg_stat_all_tables;

 custom_maintenance_work_mem | current_maintenance_work_mem 
-----------------------------+------------------------------
 5273 kB                     | 64MB
(1 row)

Il me faut au minimum 5,2 Mo. Je suis donc tranquille.

Évidemment, le nombre d'enregistrements morts évolue dans le temps et il est tout à fait possible que la quantité de mémoire nécessaire soit bien plus importante. On peut se baser sur le nombre d'enregistrements total pour avoir le pire des cas comme ici :

b1=# SELECT pg_size_pretty(max((n_live_tup+n_dead_tup)*6)) AS custom_maintenance_work_mem,
     current_setting('maintenance_work_mem') AS current_maintenance_work_mem
     FROM pg_stat_all_tables;

 custom_maintenance_work_mem | current_maintenance_work_mem 
-----------------------------+------------------------------
 472 MB                      | 512MB
(1 row)

Ce qui révèle donc une configuration adéquate pour cet utilisateur.

par Guillaume Lelarge le mardi 14 juillet 2015 à 07h54

mardi 16 juin 2015

Guillaume Lelarge

Différences entre les versions beta du livre

Je me suis rendu compte ce week-end qu'on n'avait pas publié d'informations sur ce qui avait été ajouté entre les différentes versions beta, en dehors des nouveaux chapitres. Voici donc la liste des modifications, un peu éditée pour être plus lisible :

Pour la beta2 :

  • Nouveaux chapitres
    • protocole de communication
    • connexions
  • Chapitre fichiers
    • ajout d'une note sur l'option --no-clean de la commande initdb
    • ajout d'une note sur les versions 9.1 (et inférieures) et la colonne spclocation du catalogue pg_tablespace
    • ajout d'une note sur le fichier pgstat.stat des versions 9.3 et antérieures
  • Chapitre processus
    • refonte des sections pour trier les processus par activité (et non par nom)
    • revue du résumé sur les processus d'écriture dans les fichiers de données suite à une remarque d'un lecteur dans le forum du livre (forum uniquement accessible par les lecteurs actuels)
    • correction des processus équivalents au niveau Oracle, suite là-aussi à un autre commentaire d'un lecteur
  • Chapitre mémoire
    • ajout de deux paragraphes sur l'utilisation (partielle) du cache pour les requêtes
    • présentation de deux colonnes de la vue pg_stat_database permettant de quantifier l'utilisation des fichiers temporaires

Et pour la beta 3 :

  • Nouveaux chapitres
    • gestion des objets
    • transactions
  • Global
    • remplacement du terme maître par serveur primaire et du terme esclave par serveur secondaire (suite à la demande d'un lecteur)
  • Chapitre mémoire
    • ajout d'une note sur l'intérêt du paramètre maintenance_work_mem dans le cadre d'un import de données avec pg_restore
  • Chapitre processus
    • ajout d'une partie sur les écritures dans les fichiers de données suite à un CHECKPOINT, avec quelques graphes, pour mieux expliquer les différents paramètres checkpoint_*
  • Chapitre Fichiers
    • ajout d'un paragraphe sur la génération des relfilnode
    • ajout d'infos sur le LSN et les journaux de transactions

Je publierais dans ce blog les nouveautés de chaque version beta à chaque sortie, ce sera plus facile pour les lecteurs.

par Guillaume Lelarge le mardi 16 juin 2015 à 20h32

lundi 25 mai 2015

Dimitri Fontaine

PostgreSQL au JUG de Montpellier

J'ai eu le plaisir de présenter PostgreSQL au Java User Group de Montpellier le 20 mai dernier dans le cadre d'une soirée PostgreSQL is YeSQL. L'accueil réservé par le JUG a été parfait et je tiens à les remercier pour une très bonne soirée de partage et d'échanges autour de PostgreSQL.

Les slides des présentations sont disponibles. Nous avons commencé par une présentation générale sur comment et pourquoi utiliser PostgreSQL en 2015 :

Puis nous avons continué la soirée avec une présentation des Extensions PostgreSQL, dont ce blog regorge d'exemples :

Une série d'articles sont disponibles sur ce blog et reprennent en détails les éléments présentés dans cette deuxième partie :

 

Next month, Postgres Open 2014 is happening in Chicago, and I'll have the pleasure to host a tutorial about PostgreSQL Extensions Writing & Using Postgres Extensions, and a talk aimed at developers wanting to make the best out of PostgreSQL, PostgreSQL for developers:

 

PHP Tour 2014

June, 27 2014

En début de semaine se tenait le PHP Tour 2014 à Lyon, et j'ai eu le privilège d'y être invité afin de présenter comment Utiliser PostgreSQL en 2014.

 

Last week some PostgreSQL users, contributors and advocates have organized a really great conference in Stockholm, Sweden, where I had the please to give the following talk:

 

In our previous article Aggregating NBA data, PostgreSQL vs MongoDB we spent time comparing the pretty new MongoDB Aggregation Framework with the decades old SQL aggregates. Today, let's showcase more of those SQL aggregates, producing a nice histogram right from our SQL console.

 

When reading the article Crunching 30 Years of NBA Data with MongoDB Aggregation I coulnd't help but think that we've been enjoying aggregates in SQL for 3 or 4 decades already. When using PostgreSQL it's even easy to actually add your own aggregates given the SQL command create aggregate.

 

Back from the FODESM 2014 Conference, here's the slides I've been using for the Advanced Extension Use Cases talk I gave, based on the ongoing work to be found under the Tour of Extensions index in this web site.

 

Back From Dublin

November, 05 2013

Last week I had the pleasure to present two talks at the awesome PostgreSQL Conference Europe. The first one was actually a tutorial about Writing & using Postgres Extensions where we spent 3 hours on what are PostgreSQL Extensions, what you can expect from them, and how to develop a new one. Then I also had the opportunity to present the new version of pgloader in a talk about Migrating from MySQL to PostgreSQL.

 

Denormalizing Tags

October, 24 2013

In our Tour of Extensions today's article is about advanced tag indexing. We have a great data collection to play with and our goal today is to be able to quickly find data matching a complex set of tags. So, let's find out those lastfm tracks that are tagged as blues and rhythm and blues, for instance.

 

At the Open World Forum two weeks ago I had the pleasure to meet with Colin Charles. We had a nice talk about the current state of both MariaDB and PostgreSQL, and even were both interviewed by the Open World Forum Team. The interview is now available online. Dear French readers, it's in English.

 

PostgreSQL is an all round impressive Relational DataBase Management System which implements the SQL standard (see the very useful reference page Comparison of different SQL implementations for details). PostgreSQL also provides with unique solutions in the database market and has been leading innovation for some years now. Still, there's no support for Autonomous Transactions within the server itself. Let's have a look at how to easily implement them with PL/Proxy.

 

Let's get back to our Tour of Extensions that had to be kept aside for awhile with other concerns such as last chance PostgreSQL data recovery. Now that we have a data loading tool up to the task (read about it in the Loading Geolocation Data article) we're going to be able to play with the awesome ip4r extension from RhodiumToad.

 

Last Friday I had the chance to be speaking at the Open World Forum in the NewSQL track, where we had lots of interest and excitement around the NoSQL offerings. Of course, my talk was about explaining how PostgreSQL is Web Scale with some historical background and technical examples about what this database engine is currently capable of.

 

Using trigrams against typos

September, 06 2013

In our ongoing Tour of Extensions we played with earth distance in How far is the nearest pub? then with hstore in a series about trigger, first to generalize Trigger Parameters then to enable us to Auditing Changes with Hstore. Today we are going to work with pg_trgm which is the trigrams PostgreSQL extension: its usage got seriously enhanced in recent PostgreSQL releases and it's now a poor's man Full Text Search engine.

 

In a previous article about Trigger Parameters we have been using the extension hstore in order to compute some extra field in our records, where the fields used both for the computation and for storing the results were passed in as dynamic parameters. Today we're going to see another trigger use case for hstore: we are going to record changes made to our tuples.

 

Trigger Parameters

August, 23 2013

Sometimes you want to compute values automatically at INSERT time, like for example a duration column out of a start and an end column, both timestamptz. It's easy enough to do with a BEFORE TRIGGER on your table. What's more complex is to come up with a parametrized spelling of the trigger, where you can attach the same stored procedure to any table even when the column names are different from one another.

 

There was SQL before window functions and SQL after window functions: that's how powerful this tool is. Being that of a deal breaker unfortunately means that it can be quite hard to grasp the feature. This article aims at making it crystal clear so that you can begin using it today and are able to reason about it and recognize cases where you want to be using window functions.

 

In our recent article about The Most Popular Pub Names we did have a look at how to find the pubs nearby, but didn't compute the distance in between that pub and us. That's because how to compute a distance given a position on the earth expressed as longitude and latitude is not that easy. Today, we are going to solve that problem nonetheless, thanks to PostgreSQL Extensions.

 

In his article titled The Most Popular Pub Names Ross Lawley did show us how to perform some quite interesting geographic queries against MongoDB, using some nice Open Data found at the Open Street Map project.

 

In a recent article here we've been talking about how do do Batch Updates in a very efficient way, using the Writable CTE features available in PostgreSQL 9.1. I sometime read how Common Table Expressions changed the life of fellow DBAs and developers, and would say that Writable CTE are at least the same boost again.

 

In a recent article Craig Kerstiens from Heroku did demo the really useful crosstab extension. That function allows you to pivot a table so that you can see the data from different categories in separate columns in the same row rather than in separate rows. The article from Craig is Pivoting in Postgres.

 

Tonight I had the pleasure to present a talk at the Dublin PostgreSQL User Group using remote technologies. The talk is about how to make the most ouf of PostgreSQL when using SQL as a developer, and tries to convince you to dive into mastering SQL by showing how to solve an application example all in SQL, using window functions and common table expressions.

 

Nearest Big City

May, 02 2013

In this article, we want to find the town with the greatest number of inhabitants near a given location.

 

The Need For Speed

March, 29 2013

Hier se tenait la cinquième édition de la conférence organisée par dalibo, où des intervenants extérieurs sont régulièrement invités. Le thème hier était à la fois clair et très vaste : la performance.

 

Batch Update

March, 15 2013

Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

 

HyperLogLog Unions

February, 26 2013

In the article from yesterday we talked about PostgreSQL HyperLogLog with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it.

 

PostgreSQL HyperLogLog

February, 25 2013

If you've been following along at home the newer statistics developments, you might have heard about this new State of The Art Cardinality Estimation Algorithm called HyperLogLog. This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/aggregateknowledge/postgresql-hll and soon to be in debian.

 

PostgreSQL for developers

November, 02 2012

As Guillaume says, we've been enjoying a great evening conference in Lyon 2 days ago, presenting PostgreSQL to developers. He did the first hour presenting the project and the main things you want to know to start using PostgreSQL in production, then I took the opportunity to be talking to developers to show off some SQL.

 

Reset Counter

October, 05 2012

I've been given a nice puzzle that I think is a good blog article opportunity, as it involves some thinking and window functions.

 

Let's say you need to ALTER TABLE foo ALTER COLUMN bar TYPE bigint;, and PostgreSQL is helpfully telling you that no you can't because such and such views depend on the column. The basic way to deal with that is to copy paste from the error message the names of the views involved, then prepare a script wherein you first DROP VIEW ...; then ALTER TABLE and finally CREATE VIEW again, all in the same transaction.

 

Dynamic Triggers in PLpgSQL

November, 24 2010

You certainly know that implementing dynamic triggers in PLpgSQL is impossible. But I had a very bad night, being up from as soon as 3:30 am today, so that when a developer asked me about reusing the same trigger function code from more than one table and for a dynamic column name, I didn't remember about it being impossible.

 

The drawback of hosting a static only website is, obviously, the lack of comments. What happens actually, though, is that I receive very few comments by direct mail. As I don't get another spam source to cleanup, I'm left unconvinced that's such a drawback. I still miss the low probability of seeing blog readers exchange directly, but I think a tapoueh.org mailing list would be my answer, here...

 

Window Functions example

September, 09 2010

So, when 8.4 came out there was all those comments about how getting window functions was an awesome addition. Now, it seems that a lot of people seeking for help in #postgresql just don't know what kind of problem this feature helps solving. I've already been using them in some cases here in this blog, for getting some nice overview about Partitioning: relation size per “group”.

 

Happy Numbers

August, 30 2010

After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content ( blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That's a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that's the language I use the most those days.

 

Playing with bit strings

August, 26 2010

The idea of the day ain't directly from me, I'm just helping with a very thin subpart of the problem. The problem, I can't say much about, let's just assume you want to reduce the storage of MD5 in your database, so you want to abuse bit strings. A solution to use them works fine, but the datatype is still missing some facilities, for example going from and to hexadecimal representation in text.

 

This time, we are trying to figure out where is the bulk of the data on disk. The trick is that we're using DDL partitioning, but we want a “nice” view of size per partition set. Meaning that if you have for example a parent table foo with partitions foo_201006 and foo_201007, you would want to see a single category foo containing the accumulated size of all the partitions underneath foo.

 

This time we're having a database where sequences were used, but not systematically as a default value of a given column. It's mainly an historic bad idea, but you know the usual excuse with bad ideas and bad code: the first 6 months it's experimental, after that it's historic.

 

The problem was raised this week on IRC and this time again I felt it would be a good occasion for a blog entry: how to load an XML file content into a single field?

par dim@tapoueh.org (Dimitri Fontaine) le lundi 25 mai 2015 à 15h17

lundi 18 mai 2015

Adrien Nayrat

Partition xlogs pleine

Cette semaine j’ai été confronté à un incident sur une instance postgres. La partition xlogs était pleine :

PANIC: could not write to file "pg_xlog/xlogtemp.12352": No space left on device
LOG: startup process (PID 12352) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure

Cette instance est répliquée sur un autre serveur et archive ses journaux de transaction sur un troisième serveur. Ce dernier était plein, Postgres est intelligent et a conservé ses journaux de transaction en attendant de pouvoir les archiver… Jusqu’à ce que la partition des journaux se retrouve pleine entraînant l’arrêt de Postgres.

Le nettoyage sur le serveur d’archivage n’était pas suffisant, il fallait libérer de la place sur la partition. La solution qui vient en tête est d’agrandir la partition. Je ne souhaitais pas agrandir la partition. J’ai donc fait un truc tout simple : Déplacer un fichier journal et faire un lien symbolique vers celui-ci. Postgres est reparti et j’ai lancé un « CHECKPOINT » immédiatement ce qui a libéré de la place. Ensuite il faut penser à supprimer le fichier car Postgres a juste supprimé le lien.
C’est plus rapide à mettre en place que d’agrandir la partition.

En fouillant sur google je suis tombé sur ce site :
Solving pg_xlog out of disk space problem on Postgres

L’auteur nous donne une astuce toute simple : Créer un fichier plein de vide. Si un jour la partition est pleine il suffit de le supprimer pour faire repartir l’instance :

 dd if=/dev/zero of=DO_NOT_MOVE_THIS_FILE bs=1MB count=100

Bilan : Il faut bien penser à superviser l’archivage de Postgres!

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le lundi 18 mai 2015 à 20h00

dimanche 17 mai 2015

ulhume

Rooter le LG G Pad 7.0

Lorsque l&aposon achète un device Android, se pose toujours le même problème &aposcomment vais-je rooter ce machin là&apos.

Voici donc comment faire pour la tablette LG GPad 7.0.

Pourquoi faire ?

Comme mon sujet pour l&aposannée 2015 est clairement le développement d&aposapplications mobile (allez zouh, un peu de pub ;-), après m&aposêtre commis à acheter un Mac Mini et un iPad Mini, j&aposai aussi dut faire l&aposacquisition d&aposun fairphone (pour me racheter un peu...) et d&aposune tablette Android, le LG G Pad 7.0 (le Windows Phone... m&aposa été gracieusement donné, ouf !! ;-).

Concernant les deux devices Android, seul le fairphone a le bon goût d&aposêtre rooté en sortie d&aposusine. En revanche, le LG, c&aposest prison dorée... Or pour développer sous Android, rooter est l&aposétape nécessaire. Car outre l&aposaspect philosophique (je ne supporte pas l&aposidée qu&aposon m&aposempêche de faire ce que je veux de ce que j&aposachète), l&aposaccès root me permet d&aposutiliser ADB en mode réseau (c&aposest à dire sans cable USB). Et ça, je ne sais pas m&aposen passer...

Bref, trêve de blabla, voyons comment casser la bête...

Passer en mode développeur

La première étape consiste déjà à activer le mode développeur, ce qui n&aposest pas aussi évident que l&aposon pourrait se l&aposimaginer. Pour cela vous devrez aller dans les réglages, puis dans le menu à propos de la tablette, et enfin dans information sur le logiciel. Arrivé là, accrochez-vous bien, vous devez taper plusieurs fois sur le Numéro de build.

La tablette vous demande alors si vous êtes certain de votre action (non, non, j&aposai tapé 20 fois là dessus par pur hasard...), puis fera apparaître, à la positive, le menu tant utile pour les développeurs.

Dans ce menu, vous allez devoir vous rendre dans la section debuggage pour activer le debuggage de la connection USB. Ce qui est le pré-requis indispensable pour pouvoir utiliser ADB mais aussi pour pouvoir rooter l&aposappareil.

Cas n°1, KitKat

Récupération de Purple Drake

Pour ceux qui ont acquis leur tablette à sa sortie, la version Android de base est KitKat. Pour rooter cette version, le sésame s&aposappelle Purple Drake. Purple Drake Pour le rooting à proprement parlé, le sésame s&aposappelle Purple Drake. Téléchargez donc la dernière version (R3 dans mon cas) et décompressez là en local.

Pour ceux qui utilisent (encore) Wheezy

Pour ceux qui sont sur une version récente du kernel, cette étape peut être zappée. Dans mon cas, la version de GLibC incluse dans Debian Wheezy ne me permet de lancer le script tel quel avec les versions binaires d&aposadb incluse dans l&aposarchive.

Cela se règle cependant simplement en installant la version debian d&aposadb par sudo apt-get install android-adb. Ceci fait, allez dans le dossier assets de l&aposarchive décompressée de Purple Drake et éditez le fichier purpledrake_main.sh pour remplacer en ligne 16, $1 par /usr/bin.

Lorsque tout est en ordre, il ne reste plus qu&aposà lancer l&aposoutil. Assurez-vous que la tablette n&aposa aucune application de lancée (rebootez là si nécessaire), que le cable USB est bien connecté, et que dans la barre de status de la tablette vous voyez bien que la connection USB se fait en mode debuggage.

root !

Si tout est OK, lancez le rooting par sudo ./purpledrake_linux.sh. La raison du sudo ici est que sous Debian, en standard, seul root a accès au device USB. Cela peut se configurer au niveau d&aposudev mais ça me saoule un peu d&aposavoir à faire cela à chaque fois, sur chaque machine et pour chaque device. Et c&aposest d&aposailleurs l&aposune des raisons qui me fait passer ADB en mode réseau.

Une fois l&aposoutil lancé il suffit de se laisser guider par le script. Il va d&aposabord rebooter le device, puis installer un root temporaire, puis l&aposutiliser, si vous le désirez, pour mettre en place le root permanent.

Lorsque la tablette a redémarré, tout application qui demande l&aposaccès root devrait ainsi l&aposobtenir. C&aposest une première étape mais ce n&aposest pas très sécurisé, loin de là.

Super pouvoirs à la demande

Pour aller un cran plus loin, rendez vous sur le market et téléchargez SuperSU. Cette application au démarrage va remplacer la commande su fournie par Purple Drake, par une version qui vous demandera si telle application a bien le droit d&aposobtenir l&aposaccès root.

Au premier lancement, SuperSU va vous proposer une procédure de remplacement de la commande en mode normal ou recovery. J&aposai personnellement pris l&aposoption normal.

Et après une installation avec succès suivi d&aposun redémarrage de l&aposengin, tout était opérationnel.

Félicitation, votre device vous appartient !

Cas n°2, Lollipop

Si vous avez acheté votre LG plus récemment, ou si, comme moi, vous avez clické sur "mettre à jour" sans faire tourner votre cerveau, vous n&aposêtes pas sous Kitkat mais sous Lollipop. Et là, le purple drake il marche plus du tout !

Alors heureusement il y a une autre méthode qui fonctionne parfaitement mais elle nécessite... windows. Je sais, c&aposest pas cool, mais je n&aposai pas trouvé mieux.

Pour rooter, petite liste de courses :

  1. Les pilotes Android de chez LG. Perso, je les ai trouvés .
  2. Une petite application toute mignonette qui va vous simplifier le rootage, et que vous trouverez ici

Première étape, installer les pilotes sur le windows. Je ne sais pas pour vous mais chez moi cela a pris des plombes !!!

Ceci fait, connectez votre LG par le câble USB et lancez l&aposapplication puis clickez sur ROOT DEVICE. Cela va lancer une console et le script associé va redémarrer le device. Si après ce redémarrage vous vous retrouvez à nouveau sous votre home Android, c&aposest que quelque chose n&aposaura pas marché. Dans ce cas le script va (sans doute) vous dire (je traduis ;-) "désolé, mais j&aposai échoué salement à passer le device en mode série, fait le toi-même, moi j&aposattends".

En effet, le script a besoin que le device soit dans un mode très spécial de mise à jour du firmeware qui place le port USB en mode "port série". Dans ce mode, il lui sera possible de lire un numéro de série qui permettra par la suite de rooter l&aposengin.

Pour faire cela manuellement procédez comme suit sans toucher au script côté windows qui doit afficher un waiting device. Qu&aposil attende donc...

  1. Débranchez le cable,
  2. Éteignez le device par une longue pression sur le bouton d&aposallumage, puis Éteindre,
  3. Pressez le bouton qui monte le volume (c&aposest celui qui est prêt du bouton d&aposallumage),
  4. Tout en maintenant ce bouton pressé, rebranchez le cable USB
  5. Là le device doit s&aposallumer et après un temps il va se stabiliser dans un mode étrange avec une sorte de barre de progression pour la mise à jour du firmware.
  6. Le script côté windows doit logiquement se réveiller et recommencer à vous causer,
  7. Lorsque le script a terminé son boulot, le device a redémarré en mode Android classique.
  8. Le script vous demande (en anglais) de l&aposarrêter (???) en pressant les touches Control+C , puis de pressez N puis Enter.
  9. Ceci fait, vous êtes de retours sur l&aposinterface graphique.

À ce stage, normalement c&aposest tout bon et le LG est rooté. Pour tester, déverrouillez la home et sur l&aposinterface côté Windows pressez le bouton "ADB SHELL". Vous devriez voir apparaître une console avec quelque chose comme &aposshell@e7wifi:/ $&apos. Maintenant tapez su puis validez. Et là, côté Android, une boite de dialogue doit apparaître pour vous demander si vous authoriser ce passage en mode root. Acceptez. De retour sous Windows, dans la console, devrait alors s&aposafficher root@e7wifi:/ #.

C&aposest bon, vous êtes root !

Tester le tout

Pour tester, le plus simple est d&aposinstaller ADB Wifi à partir du market et de le lancer. Si les étapes précédentes ont fonctionnées, lorsque vous activerez la ADB en mode réseau, une boite de dialogue doit apparaître pour valider l&aposaccès.

Ceci fait, ADB Wifi doit vous indiquer qu&aposil est en écoute et vous fournis l&aposIP de connection. Vous n&aposavez alors plus qu&aposà vous connecter de votre machine comme ceci :

$sudo adb connect 192.168.154.21
connected to 192.168.154.21:5555
$sudo adb shell

Conclusion

Il est toujours navrant d&aposavoir à perdre du temps sur quelque chose d&aposaussi trivial.

Espérons que peu à peu les constructeurs s&aposinspire de FairPhone et arrêtent ainsi de prendre leurs clients pour des crétins... On peut toujours rêver ;-)

Please enable JavaScript to view the comments powered by Disqus.

dimanche 17 mai 2015 à 22h13