Administering PennGroups Chris Hyzer ISC/ASTT August 1, 2011 8/24/2015ISC1.

Administering PennGroupsAdministering PennGroups


Chris HyzerChris Hyzer

ISC/ASTTISC/ASTT

August 1, 2011August 1, 2011

04/19/23 ISC 1



Install– DDL– API– UI– WS

Source control Upgrade Maintenance Roadmap

04/19/23 ISC 2


Demo serverDemo server

Internet2 has a Grouper Demo Server Address is: https://grouperdemo.internet2.edu/ Host various versions of Grouper Show features (e.g. permissions, external users,

syncing between groupers) Allow users or potential users to kick the tires (not for

production obviously)


Demo server for this trainingDemo server for this training

9 accounts setup /home/test1, /home/test2, etc Each participant gets a temporary account and pass

(ask Chris) The pass will work for SSH, Apache (UI, WS), mysql Each account has a mysql database, and user with

access to that database


Demo server training tomcatsDemo server training tomcats

Except for development, you should only have one application per Tomcat– One application wont take all memory and kill tomcat– You can restart after deploys without affecting other apps

Note: for development, you can use either basic authentication– Or link cosign/shib tomcat on test cluster to dev tomcat


Demo server training tomcats Demo server training tomcats (continued)(continued)

Install dir:[test6@i2midev1 tomcat]$ pwd/home/test6/tomcat[test6@i2midev1 tomcat]$ lsconf logs temp webapps work

TOMCAT_HOME, CATALINA_HOME elsewhere[test6@i2midev1 tomcat]$ more \ /etc/init.d/tomcat_test6…export TOMCAT_HOME="/opt/tomcat6base" export CATALINA_HOME="/opt/tomcat6base"



Note you have your own Java symlink (for ps)

[test6@i2midev1 tomcat]$ more /etc/init.d/tomcat_test6…export JAVA_HOME="/opt/javas/java_test6"

Start/stop tomcat, lets run these commands (for your user)

[test6@i2midev1 ~]$ tomcat start[test6@i2midev1 ~]$ ps -ef | grep java_test6[test6@i2midev1 ~]$ netstat -pan | grep 5231

(note the port numbers)[test6@i2midev1 ~]$ jstack 5231 | less

[test6@i2midev1 ~]$ tomcat stop[test6@i2midev1 ~]$ ps -ef | grep java_test6



To set this up:$ sudo useradd -g i2mi test1$ sudo passwd test1 - create user in DB, database assign permissions with sql yogmysql> create database grouper;mysql> create user 'grouper'@'localhost' identified by

'somesecret';mysql> grant all on grouper.* to 'grouper'@'localhost';$ sudo htpasswd /etc/httpd/conf.d/users.pass test1$ sudo htpasswd /etc/httpd/conf.d/users.pass test1a$ cp -R /opt/tomcats/tomcat6bullet /home/test6/tomcatports start at 9010, edit the ~/tomcat/conf/server.xml, set the three ports:

http: 90x0, jk: 90x1, shutdown: 90x2[root@i2midev1 init.d]# cp tomcat_k tomcat_test1[root@i2midev1 init.d]# chkconfig --add tomcat_test1[appadmin@i2midev1 javas]$ ln -s ../java6 java_test1[test1@i2midev1 test1]$ mkdir bin[test1@i2midev1 bin]$ ln -s /etc/init.d/tomcat_test1

tomcat



Put this in /etc/profile

JAVA_HOME=/opt/java6export JAVA_HOME

ANT_HOME=/opt/antexport ANT_HOME

#note: maven3 is needed for grouper 2.0M2_HOME=/opt/mavenexport M2_HOME

M2=$M2_HOME/binexport M2

PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$M2:$PATH:$HOME/binexport PATH



Put this in /var/www/html<b>test1</b><ul> <li><a href="test1_grouper/">Grouper UI</a></li> <li><a href="test1_grouperWs/">Grouper WS</a></li></ul> Put this in /etc/httpd/conf.d/proxy_ajp.confProxyPass /test1_grouper/ ajp://localhost:9011/test1_grouper/ProxyPass /test1_grouperWs/ ajp://localhost:9011/test1_grouperWs/

Bounce apache[root@i2midev1 init.d]# /sbin/service httpd

configtestSyntax OK[root@i2midev1 init.d]# /sbin/service httpd

graceful[root@i2midev1 init.d]#


Install Grouper API (First step in training)Install Grouper API (First step in training)

Download the API (1.6.3, latest stable)$ mkdir 1.6.3$ cd 1.6.3$ pwd/home/test1/1.6.3$ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.apiBinary-1.6.3.tar.gz

$ tar xzvf grouper.apiBinary-1.6.3.tar.gz$ cd grouper.apiBinary-1.6.3/conf/$ emacs grouper.properties


Install Grouper API (continued)Install Grouper API (continued)

Note, Powerpoint might mess-up chars (e.g. dashes)… might need to type them in if there is a problem, instead of copy/paste

Note, a backslash: \ means the next line is the same line Note, all commands are in the "notes" section of the

presentation, copy/paste from there


Emacs cheatsheetEmacs cheatsheet

Note: Feel free to use your favorite editor obviously

$ emacs somefile.whateverNote: might need to do CTRL-backspace instead of backspaceSave: CTRL-x CTRL-sExit: CTRL-x CTRL-cFind: CTRL-sFind (wrap): CTRL-s, put in criteria, CTRL-sFind backwards: CTRL-rStop command: CTRL-gReplace: ESC-x query-replace (then “y” to replace each)Cut rest of line: CTRL-kPaste: CTRL-yPut in background: CTRL-zBASH get back: fg


Configure grouper.propertiesConfigure grouper.properties

$ emacs grouper.propertiesgroups.wheel.use = true

configuration.autocreate.system.groups = true configuration.autocreate.group.name.0 = etc:webServiceClientUsers configuration.autocreate.group.description.0 = users allowed in WSconfiguration.autocreate.group.subjects.0 = GrouperSystem,testX,testXaconfiguration.autocreate.group.name.1 = etc:sysadmingroup configuration.autocreate.group.description.1 = sys admin users configuration.autocreate.group.subjects.1 = testX

grouperIncludeExclude.use = truegrouperIncludeExclude.requireGroups.use = true


Configure Configure grouper.hibernate.propertiesgrouper.hibernate.properties

$ emacs grouper.hibernate.propertieshibernate.dialect = org.hibernate.dialect.MySQL5Dialect hibernate.connection.driver_class = com.mysql.jdbc.Driverhibernate.connection.url = jdbc:mysql://localhost:3306/testX hibernate.connection.username = testXhibernate.connection.password = *******************


Configure grouper-loader.propertiesConfigure grouper-loader.properties

$ emacs grouper-loader.propertiesloader.autoadd.typesAttributes = true


Init DB and startup GSHInit DB and startup GSH

$ mysql -utest1 -p test1mysql> show tables;Empty set (0.00 sec)mysql> exitBye$ cd /home/testX/1.6.3/grouper.apiBinary-1.6.3/bin/$ ./gsh.sh -registry<Shows menu>$ ./gsh.sh -registry -check -runscript$ ./gsh.sh -registry -check$ mysql -utestX -p testXmysql> show tables;76 rows in set (0.00 sec)mysql> exitBye


Quickstart and subjectsQuickstart and subjects

$ wget -O quickstart.xml http://anonsvn.internet2.edu/cgi-\ bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\ builder/quickstart.xml?view=co$ wget -O subjects.sql http://anonsvn.internet2.edu/cgi-\ bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\ builder/subjects.sql?view=co$ ./gsh.sh -registry -runsqlfile subjects.sql$ ./gsh.sh -xmlimportold GrouperSystem quickstart.xml


MySQL GUIMySQL GUI

Optional (if you have SQLYog free gui, or whatever)Port forward over SSH, remote port 3306, local port whatever (3302?)Connect to localhost 3302

In either case, lets open a GSH window, a mysql window, and a linux window


Add your usersAdd your users

$ ./gsh.sh(note: testX was automatically inserted and into wheel and WS)gsh 0% grouperSession = GrouperSession.startRootSession();gsh 1% addSubject("testX", "person", "John Smith");gsh 2% addSubject("testXa", "person", "Johna Smitha");gsh 3% addMember("etc:webServiceClientUsers", "testXa");gsh 4% exit

Review other GSH commandsNote: GrouperSession is a ThreadLocal ActAs for the API


Add your users (continued)Add your users (continued)

$ mysql -utestX -p testXINSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\

VALUES ('testX', 'loginid', 'testX', 'testX');INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'name', 'John Smith', 'john smith');INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\

VALUES ('testX', 'description', 'John Smith', 'john smith');INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\

VALUES ('testXa', 'loginid', 'testXa', 'testXa');INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\

VALUES ('testXa', 'name', 'Johna Smitha', 'johna smitha');INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\

VALUES ('testXa', 'description', 'Johna Smitha', 'johna smitha');COMMIT;exit;


Get the UIGet the UI

$ cd ~/1.6.3$ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ui-\ 1.6.3.tar.gz$ tar xzvf grouper.ui-1.6.3.tar.gz$ cd grouper.ui-1.6.3/$ cp build.properties.template build.properties$ emacs build.propertiesgrouper.folder=../grouper.apiBinary-1.6.3should.copy.context.xml.to.metainf=falsewebapp.name=testX_grouperdefault.webapp.folder=/home/testX/tomcat/webapps/${webapp.name}


Get the UI (continued)Get the UI (continued)

$ emacs webapp/WEB-INF/web.core.xmlFrom the bottom, remove security-contraint,login-config,security-roleNote, if you are running on tomcat locally, leave that in, and edit

tomcat-users.xml$ ant default$ tomcat restartGo to: https://grouperdemo.internet2.edu and click on your UI:https://grouperdemo.internet2.edu/testX_grouper/


UI authenticationUI authentication

In this case, I added this to the httpd.conf or include:<LocationMatch ^/test.*> AuthType Basic AuthName "By Invitation Only" AuthUserFile /etc/httpd/conf.d/users.pass

Require valid-user</LocationMatch>

Bounce apache:[root@i2midev1 conf.d]# /sbin/service httpd configtestSyntax OK[root@i2midev1 conf.d]# /sbin/service httpd graceful

Authentication is pluggable, can write a servlet filter to put in remote user

Works with Shib, CAS, Cosign, Basic (web server), Basic (app server), any web server plugin, out of the box

Note: if UI is internal users, but using external users via invites, might map multiple URLs to multiple authn schemes (like Penn or demo server)


Create objectsCreate objects

Act as admin in upper right. Create a root folder named "test". (system and friendly) Inside that folder (aka stem), create a group called

"testGroup", but uncheck allow all to "read" and "view". (system and friendly name is testGroup)

Add some members to testGroup: babl, babr, babu


Get the WSGet the WS

$ cd ~/1.6.3$ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ws-\ 1.6.3.tar.gz$ tar xzvf grouper.ws-1.6.3.tar.gz$ cd grouper.ws-1.6.3/grouper-ws$ emacs build.propertiesgrouper.dir=../../grouper.apiBinary-1.6.3webapp.name=testX_grouperWs$ emacs conf/grouper-ws.propertiesws.client.user.group.name = etc:webServiceClientUsers$ emacs webapp/WEB-INF/web.xmlFrom the bottom, remove security-contraint,login-config,security-roleNote, if you are running on tomcat locally, leave that in, and edit

tomcat-users.xml


Get the WS (continued)Get the WS (continued)

$ ant dist$ cp -R build/dist/testX_grouperWs ~/tomcat/webapps$ tomcat restartGo to: https://grouperdemo.internet2.edu and click on your WS:https://grouperdemo.internet2.edu/testX_grouperWs/servicesRestShould get error though there is a URL to get members…


WS authenticationWS authentication

In this case, it’s the same as UI:<LocationMatch ^/test.*> AuthType Basic AuthName "By Invitation Only" AuthUserFile /etc/httpd/conf.d/users.pass

Require valid-user</LocationMatch>

Bounce apache:[root@i2midev1 conf.d]# /sbin/service httpd configtestSyntax OK[root@i2midev1 conf.d]# /sbin/service httpd graceful

Authentication is pluggable, or can write a servlet filter to put in remote user

Works with Kerberos, SOAP WS-security, Basic (web server), Basic (app server), any web server plugin, out of the box


Loader exampleLoader example

Lets make a table with subjectIds in it

mysql> CREATE OR REPLACE VIEW loader_employee AS \ (SELECT subjectId AS subject_id FROM testX.subject\ WHERE subjectId LIKE 'b%');

Make a folder for community under rootAdd a group called employee under folder "community"Edit the employee group, select type "grouperLoader" (make sure acting as admin in upper right of UI)


Loader example (continued)Loader example (continued)

Edit attributes on the group:grouperLoaderDbName: grouperNOTE: configure other DB connections in grouper-loader.properties

NOTE: every minute just for testing…grouperLoaderQuartzCron: 0 * * * * ? grouperLoaderQuery: select subject_id subject_id from \ loader_employeegrouperLoaderScheduleType: CRONgrouperLoaderType: SQL_SIMPLE



Run manually just to see it work:gsh 0% grouperSession = GrouperSession.startRootSession();gsh 1% loaderGroup = GroupFinder.findByName(grouperSession, \"community:employee");gsh 2% loaderRunOneJob(loaderGroup);loader ran successfully, inserted 12 memberships, deleted 0 memberships, total membership count: 12gsh 3%



Run continuously:% ./gsh.sh -loader

Change the view:mysql> CREATE OR REPLACE VIEW loader_employee AS \ (SELECT subjectId AS subject_id FROM SUBJECT \ WHERE subjectId LIKE 'b%o' or subjectId like 'el%');

At the top of the minute, check the memberships, and there should be some deleted, and some added


Loader details at PennLoader details at Penn

In the fastGrouperProdDaemon web application, we run the loader jobs in a FAST daemonAll loader jobs are based on views, to ease maintenanceThen we don’t have a command line application to monitor etc.The grouper_loader_log table has an entry for every daemon runGenerally the only problem we have is Warehouse jobs for people without PennId’s, the people can’t be found, and it’s an error

–The warehouse will assign a fake pennId which starts with 0–Change the loader view to have where clause PENN_ID not like '0%'


Loader include/exclude exampleLoader include/exclude exampleCreate a groupRead/update should not be granted to everyoneUse addIncludeExclude type

Look in folder, there will be 5 groups created with that type.Open the system of record, and lets make that the loader group

Create this view in the DB:mysql> CREATE OR REPLACE VIEW loader_student AS \(SELECT subjectId AS subject_id FROM SUBJECT WHERE \subjectId LIKE 'fi%');


Loader include/exclude example (continued)Loader include/exclude example (continued)

Never edit the loader group, unless you expect it to get overwrittenAdd fico to the excludes groupAdd bapo to the includes groupLook at the overall groupGenerally the privileges are:Assign READ on all to adminsAssign UPDATE on include/exclude groups to adminsAssign READ to service principal of app for overall group or other people who need to use the group


Customize UI textCustomize UI text

% cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources% mkdir custom% cd custom% touch media.properties% emacs nav.propertiesfind.browse.here=Current folder is:% emacs init.propertiesdefault.module=grouperdefault.locale=en_US% cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources% emacs init.propertiesdefault.module=customdefault.locale=en_USBounce tomcat: % tomcat restart


Customize UI text (continued)Customize UI text (continued)

Should see:


Customize lite UI for an applicationCustomize lite UI for an application

% cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes% mkdir membershipLiteName% cd membershipLiteName% touch testName.properties% cd ~/tomcat/webapps/testX_grouper/WEB-\INF/classes/resources% emacs custom/nav.propertiesAdd line:membershipLiteName.testName.simpleMembershipUpdate.updateTitle = PTO\ admins

Bounce tomcat: % tomcat restart

https://grouperdemo.internet2.edu/testX_grouper/grouper\Ui/appHtml/grouper.html?operation=SimpleMembership\Update.init&groupName=apps:pto:ptoAdmins_systemOfRecord\&membershipLiteName=testName


Customize lite UI for an application (continued)Customize lite UI for an application (continued)


Get the Grouper ClientGet the Grouper Client$ cd ~/1.6.3$ wget \ http://www.internet2.edu/grouper/release/1.6.3/ \ grouper.clientBinary-1.6.3.tar.gz$ tar xzvf grouper.clientBinary-1.6.3.tar.gz$ cd grouper.clientBinary-1.6.3$ emacs grouper.client.propertiesgrouperClient.webService.url = \

https://grouperdemo.internet2.edu/testX_grouperWs/servicesRestgrouperClient.webService.login = testXgrouperClient.webService.password = **************


Get the Grouper Client (continued)Get the Grouper Client (continued)Get usage:$ java -jar grouperClient.jar$ java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup

Customize the output:$ java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup \ --outputTemplate='${wsSubject.id}$newline$'


Try from your local machine (win, mac, etc)Try from your local machine (win, mac, etc)

NOTE: you need java6+> cd c:\temp (or translate for mac or whatever)Download and unzip:

http://www.internet2.edu/grouper/release/1.6.3/grouper.clientBinary-1.6.3.tar.gz

> cd grouper.clientBinary-1.6.3Edit grouper.client.propertiesgrouperClient.webService.url = \

https://grouperdemo.internet2.edu/testX_grouperWs/servicesRestgrouperClient.webService.login = testXgrouperClient.webService.password = **************

Get usage:> java -jar grouperClient.jar> java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup


Grouper deployment control at PennGrouper deployment control at Penn

See document The Grouper team hopes to have a maven version of

this some time soon Everything in Penn’s CVS, external encrypted

passwords, separated out filters, customizations Localdev, Dev, Test, Prod managed for custom

application, WS, UI, client Checkout pennGrouper project from CVS and look at

customizations


Grouper upgrades at PennGrouper upgrades at Penn

See document of 1.6 to 1.7 upgrade I believe we skipped 1.5, though we generally

upgrade to new versions when available since we need features

Generally we want everything to be up as much as possible in readonly mode

Switch to readonly mode for WS/UI, stop updates on LDAP

Get counts of important groups for sanity test Upgrade the DB (generally most time consuming

part, if have to edit millions of rows) Upgrade the WS/UI, test everything Confirm counts of important groups


Grouper maintenance at PennGrouper maintenance at Penn

Process forms from DA, add kerberos principals Add new loader jobs Add new folders and delegate to new clients Consult on design of how applications use Grouper Look at errors emailed from Grouper log4j Run GSH scripts


Grouper sample GSH scriptGrouper sample GSH script

See this documentation E.g. will need to do this to delete old course groups


Daily reportDaily report

Email sent everyday to give a pulse of Grouper Need to deal with unresolvable subjects


MonitoringMonitoring

grouperWs has a status servlet hooked up to nagios /grouperWs/status?diagnosticType=all


RoadmapRoadmap

Hopefully uses for central permissions– E.g. warehouse permissions– E.g. PennCommunity Direct permissions

Always available read-only web services Shibboleth entitlement group membership integration PennCommunity Direct getPerson WS secure

attributes


Questions?Questions?

04/19/23 ISC 50

Administering PennGroups Chris Hyzer ISC/ASTT August 1, 2011 8/24/2015ISC1.

Documents

Transcript of Administering PennGroups Chris Hyzer ISC/ASTT August 1, 2011 8/24/2015ISC1.