remove duplicate lines in each paragraph of a file using sed or awk -

-

i want remove duplicate lines pargraphs begin "set current" in file, share same first line , have same sentences , don't remove duplicate lines belong different paragraphs example:

if have following file:

set current = 'aaa' ; create syn file1 1000.file1 ; create syn file2 1000.file2 ; create syn file3 1001.file3 ; create syn file3 1001.file3 ;  set current = 'aaa' ; create syn file1 1000.file1 ; create syn file2 1000.file2 ; create syn file7 1000.file7 ;  set current = 'bbb' ; create syn file5 1002.file5 ; create syn file6 1003.file6 ;  set current = 'bbb' ;   create syn file1 1000.file1 ; create syn file8 1002.file8 ; create syn file6 1003.file6 ;

the result 

set current = 'aaa' ; create syn file1 1000.file1 ; create syn file2 1000.file2 ; create syn file3 1001.file3 ;  set current = 'aaa' ; create syn file7 1000.file7 ;  set current = 'bbb' ; create syn file5 1002.file5 ; create syn file6 1003.file6 ;  set current = 'bbb' ; create syn file1 1000.file1 ; create syn file8 1002.file8 ;

with awk this:

awk 'nf==0{print;next};/^set current/{c=$4;print;next}!seen[c,$0]++' file

with comments make more readable:

awk ' nf == 0 {       # if find empty line           print       # print line           next        # , skip next record       }       /^set current/{ # if find line beginning wiith "set current"           c = $4      # store value in 4th field           print       # print current line           next        # , skip next record         }       !seen[c,$0]++  # print if combination of "c" value                       # , current line has not been stored                        # in array "seen", , store                       # combination in array                       # (in order prevent other lines printed)       ' file

the !seen[c,$0]++ works this: when use comma in array index, 2 tokens combined single string joined subsep character. in case use index combination of c character , current line ($0), since needs unique after filtering. !seen[c,$0] check see if combination exists index array. if index not present, expression evaluates true, results line being printed. if index present, expression evaluates false, , line not printed. post-fix increment operator count occurrences of index, line printed @ first occurrence, not subsequent matches.


Comments

Post a Comment

Popular posts from this blog

java - Run spring boot application error: Cannot instantiate interface org.springframework.context.ApplicationListener -

-
i have spring project , try make use spring boot , run @ embadded tomcat following : https://spring.io/guides/gs/rest-service/ this application //@configuration //@enableaspectjautoproxy @springbootapplication @componentscan(basepackages = "gux.prome") public class application { public static void main(string[] args) { springapplication.run(application.class, args); } } if use maven command: mvn spring-boot:run , project starts fine, need debug , run main method @ intelij, exception occurs: exception in thread "main" java.lang.illegalargumentexception: cannot instantiate interface org.springframework.context.applicationlistener : org.springframework.boot.logging.classpathloggingapplicationlistener @ org.springframework.boot.springapplication.createspringfactoriesinstances(springapplication.java:414) @ org.springframework.boot.springapplication.getspringfactoriesinstances(springapplication.java:394) @ org.springframework.bo...
Read more

python - pip wont install .WHL files -

-
when try install .whl gohike typing cmd c:\users\owner\downloads>pip install ta_lib-0.4.9-cp27-none-win_amd64.whl i print out: `processing c:\users\owner\downloads\ta_lib-0.4.9-cp27-none-win_amd64.whl complete output command python setup.py egg_info: traceback <most recent call last>: file "<string>", line 1, in <module> ioerror: [errno 2] no such file or directory:'c:\\users\\owner\\appdata\\lo cal\\temp\\pip-9gwk2c-build\\setup.py' ----------------------------------------- command "python setup.py egg_info" failed error code 1 in c:\users\owner\ap pdata\local\temp\pip-9gwk2c-build\` the part between pip- , -build in pip-9gwk2c-build different on same file. i asked python 2.7.11 pip 8.0.3 can't install gohike whl of ta-lib 0.4.9 few days ago , decided ask differently. can me! you have outdated pip not understand wheels. wheels not supposed have setup.py inside. try upgrading pip pip in...
Read more

Excel VBA "Microsoft Windows Common Controls 6.0 (SP6)" Location Changes -

-
i have excel workbook used generate quotations customers. there main page user inputs data, , there buttons generate various pdfs , send them customer. however, 1 user consistently getting error preventing generating , sending pdfs. i beleive "microsoft windows common controls 6.0 (sp6)". it's installed on users' computers. on mine, location "c:\windows\syswow64\mscomctl.ocx" on others "c:\windows\system32\mscomctl.ocx" the user experiencing issues on 32-bit machine, "mscomctl.ocx" file should in system32. have manually put in there, , pointed common controls reference in vba it. issue seems keep reverting looking in syswow64. since folder doesn't exist, can't find it, , can't produce pdfs. i thought perhaps issue user running older version of office (2007) there users on 2013 , 2016. thought perhaps mixture of 64-bit , 32-bit users. user gets sent workbook other users (as quotation process goes further, gets passed ...
Read more